Roadmap for production readiness of various features

Help
1

We are considering using ipfs in production for a commercial product. This would be a private cluster, completely separate from the public network. Is there a roadmap for when various features will enter “production ready” status?

E.g. it is clear that pubsub is very far from being production ready. That is why we are just going to go with a centralised alternative for now. We would only rely on adding and retrieving blocks and maybe IPNS.

However it seems that there are even features related to this basic functionality that are not production ready yet. For example, garbage collection of non-pinned blocks is disabled by default because it is too slow. And there are various forum posts complaining about incredibly slow IPNS operations, which matches my experience from cursory examination.

It would be extremely helpful to have a roadmap when which feature will reach “production ready” status.

4 Likes
2

I think we are far away from production ready in general: IPFS is “eating” a huge amount of memory and CPU.

I have heard that the roadmap will be connected to Filecoin ICO… meaning that part of the money going into ProtocolLabs from the ICO will help also IPFS development (please somebody correct me if I am wrong).

Said that, I could say that we need to wait longer to have people hired and fixing bugs/problems. It may take time.

3 Likes
3

I started contributing to IPFS like 2 years ago. At the time, Juan had like $1M in funding from YC. At the time, I referred to IPFS as the “$20M project with a $1M budget”. In the grand scheme of things, IPFS + Filecoin are well worth the $20M, but they needed to get the funding.

As of this recent ICO, I think they finally have the funding.but they’re still going to need some serious ramp-up. The go-ipfs library has a limited number of recent committers, the JS library is managed almost entirely by a single person for most of 2017.

If you have a business that can financially help with IPFS or provide technical resources to improve these features, then you should find a way to talk to Juan. They do regular calls with contributors and they maintain all of their “Project Management” stuff here.

3 Likes
4

We are a startup ourselves, so we are not in a position to help financially. We would be able to commit technical resources if we see this as a way forward.

The initial use case we are targeting is very basic, so we would be able to deal with less-than-optimal performance. Also, we are willing to live on the bleeding edge and suffer, if it gives us a big competitive advantage later. I am convinced that something like ipfs is the right way to go for our application in the long term.

The only thing I would like to see is some indication how and when a part of the money from the filecoin ICO is going to be used to make the core features of IPFS rock solid and fast. IPFS, IPNS and later pubsub is all we need.

1 Like
5

The Filecoin token sale is open for another 8 days. This is just a guess, but I’d say it’s safe to expect some news shortly after the sale ends.

Personally I’m hoping this new source of funding lets Protocol Labs take a bet or two on a flagship product/service of their own making, akin to Google’s Nexus phone.

Another, waaay more outlandish hope that I’m clinging on to, would be for Protocol Labs to acquire the Matrix / Riot.im team.

  • also in the business of decentralized technology
  • (newest) tech is built with Go
  • experts on cryptography, but from a different viewpoint, which makes for healthy cross-pollination
  • considering cryptocurrencies
  • in need of funding
  • already expressed willingness to collaborate with the IPFS team
  • Orbit, while a novel project, is still very much in the experimental stages. Matrix works right now and has considerable adoption.
1 Like
6

Hey everyone,
To answer a few questions from the OP:

For example, garbage collection of non-pinned blocks is disabled by default

Its not disabled by default for performance reasons, we have this enabled on all the ipfs.io gateways. It’s disabled by default due to the possibility of it triggering while users are in the middle of working with some ipfs objects that they havent pinned yet. We will be setting this to be on by default when we have something like a journaled blockstore that will allow users to not worry about blocks they are working with randomly getting garbage collected.

And there are various forum posts complaining about incredibly slow IPNS operations

Yeah, we’re working on this. We have an IPNS publisher and resolver that uses pubsub for all of its operations: namesys/pubsub: pubsub Publisher and Resolver by vyzo · Pull Request #4047 · ipfs/kubo · GitHub Its not finished yet, but anyone who wants to help push that forward should go try it out, review the code, comment, ask questions, etc.

The initial use case we are targeting is very basic

I would love to hear more about the usecase, knowing more usecases helps us to more effectively direct our work.

The only thing I would like to see is some indication how and when a part of the money from the filecoin ICO is going to be used to make the core features of IPFS rock solid and fast. IPFS, IPNS and later pubsub is all we need.

Two things right off the bat are hiring, and setting up security audits of the codebase. Making IPNS fast and pubsub resilient are high priorities too, its hard work, but expect to see significant improvements there in the next several months.

I think its also worth mentioning a few of our new active committers, @Magik6k @stebalien and @vyzo have been doing an awesome job pushing things forward and really making a noticeable difference in the performance of ipfs. The upcoming 0.4.11 release is noticeably better on many fronts.

Anyways, Let me know if you have any other specific questions or if I have missed anything.

5 Likes
7

Thanks for taking time to answer my questions

Its not disabled by default for performance reasons, we have this enabled on all the ipfs.io gateways. It’s disabled by default due to the possibility of it triggering while users are in the middle of working with some ipfs objects that they havent pinned yet. We will be setting this to be on by default when we have something like a journaled blockstore that will allow users to not worry about blocks they are working with randomly getting garbage collected.

Thanks for the explanation. Makes sense.

Yeah, we’re working on this. We have an IPNS publisher and resolver that uses pubsub for all of its operations: https://github.com/ipfs/go-ipfs/pull/40471 Its not finished yet, but anyone who wants to help push that forward should go try it out, review the code, comment, ask questions, etc.

Sounds good. Depending on pubsub would require pubsub to move out of the experimental state, right?

I would love to hear more about the usecase, knowing more usecases helps us to more effectively direct our work.

We want to use ipfs as infrastructure to write applications in an industrial environment. We want to avoid having a single point of failure, and also tolerate network partitions. Target devices would include android devices.

One core application is storage and dissemination of event logs in an event sourcing system (kind of similar to what orbit-db does internally). Initially, the rate of event generation would be very low, so we could live with less-than-optimal performance.

Other applications are distribution of static and infrequently changing assets, which would be really easy to implement with just IPFS and IPNS.

I could give you more details privately.

I think its also worth mentioning a few of our new active committers, @Magik6k @stebalien and @vyzo have been doing an awesome job pushing things forward and really making a noticeable difference in the performance of ipfs. The upcoming 0.4.11 release is noticeably better on many fronts.

I will definitely check out if 0.4.11 fixes some of the issues I have had.

8

Hi all,

We’ve come a long way since we released the IPFS Alpha in February 2015. A ton of new features, bug fixes, performance improvements have been shipped in a joint effort with the whole community. It has been very humbling and exciting to work with all of our contributors (and wow there are so many now!).

Expect a big update on each project Roadmap (go-ipfs, js-ipfs, multiformats, IPLD, formats, libp2p and so on) that will include features and testing infrastructure to ensure the stability and performance keep increasing. We pay a lot of attention to the issues reported and features requested to help us define each project roadmap, if you are blocked by a missing feature, a bug or even a piece of documentation, make sure to open an issue, open a thread here in discuss or even send us an email to let us know (we tend to prefer Github).

We are also investing resources on getting Security Audits to the whole code base and we are also working on completing the protocol specification so that it is up to date.

In terms of activity, it might be hard to grasp across so many modules, but there are multiple people shipping code in js land, some honourable mentions go for:

  • @dignifiedquire that contributes to the whole stack, having created some of the core pieces from scratch (Bitswap, Secio, DHT) and also creating our build tool, aegir, that saves countless hours and puts every module to the same level of rigor when it comes to testing in Node.js, Browser and WebWorkers!
  • @pgte for improving a ton the unixfs-engine, implementing sharding and leading the whole charge on CRDT land and js-ipfs
  • @kumavis for building all the IPLD Ethereum adapters. Thanks to kumavis, js-ipfs has support for ethereum through the dag API
  • @VictorBjelkholm for contributing to the whole stack and now taking the lead of deploying and monitoring js-ipfs nodes in the same way that we do in go-ipfs
  • @wanderer who is pushing IPLD implementation, improving performance and creating a lot of modules to make its usage way nicer
  • @magik6k who is now getting started to contribute by implementing js-ipld-git
  • @dmitriy who is pushing the whole endeavour of Circuit Relay in js-libp2p + js-ipfs land
  • @fbaiodias who implemented a lot of the endpoints for the http-api in js-ipfs land
  • @nunofmn who refactored the whole js-ipfs-api to make it modular, a feature long requested
  • So many others! Actually, I would love to have a better way to show appreciation for every contribution in every release. If you have time to code a tool that can go over many repos and grab usernames and commits shipped between two releases, I want your help :slight_smile:

Btw, everyone is welcome to join our Monday All Hands https://github.com/ipfs/pm/issues/491 where every week multiple endeavours within the IPFS stack or on top of the IPFS stack are discussed. We now also have a dedicated call for IPLD (ipld.io) every thursday (Today!) https://github.com/ipfs/pm/issues/484.

5 Likes
9

@DeanMarkTaylor used Google Big Query to list all Discourse contributors for a specific release timeframe:

Preamble

Mostly covered by this: Google Cloud コンソールを使用して一般公開データセットに対してクエリを実行する  |  BigQuery

  1. Have Google account
  2. Create Project in Google Console: https://console.cloud.google.com/
  3. Enable Big Query API for project: https://console.cloud.google.com/apis/api/bigquery/overview
  4. Go to Big Query UI: https://bigquery.cloud.google.com/
  5. Hit Compose Query

The Query to Execute

SELECT
  repo.name,
  JSON_EXTRACT_SCALAR(payload, '$.pull_request.user.login') as user_login,
  count(*) as pull_request_merged_count,
  FORMAT_UTC_USEC(MIN(created_at)) AS first_pull_accepted_at,
FROM ( TABLE_DATE_RANGE([githubarchive:day.], TIMESTAMP('2016-04-02'), TIMESTAMP('2016-08-04')) )
WHERE
  type='PullRequestEvent'
  AND LEFT(repo.name, LENGTH("discourse/")) = "discourse/"
  AND created_at >= PARSE_UTC_USEC('2016-04-01 23:59:59')
  AND created_at < PARSE_UTC_USEC('2016-08-04 23:59:59')
  AND JSON_EXTRACT_SCALAR(payload, '$.pull_request.user.login') NOT IN ('ZogStriP', 'coding-horror', 'eviltrout', 'nlalonde', 'SamSaffron', 'techAPJ', 'tgxworld', 'riking', 'oblakeerickson', 'mpalmer', 'awesomerobot', 'Supermathie', 'xfalcox', 'nicksahler')
  AND JSON_EXTRACT_SCALAR(payload, '$.action') = "closed"
  AND JSON_EXTRACT(payload, '$.pull_request.merged') = "true"
GROUP BY
  user_login,
  repo.name
ORDER BY
  repo.name,
  pull_request_merged_count DESC,
  first_pull_accepted_at,
  user_login

Costs

First 1 TB of queries per month is free, $5 per TB after that. Executing this query once cost 236GB of my allowance - be aware if developing and executing multiple times quickly adds up.

I’m actually thinking a self-executing Google Spreadsheet would be easy to complete which outputs analysis text ready for insertion into post.

2 Likes