If I have ~10mb files I've added to IPFS, and those files are logs comprised of thousands of individual log messages, is there a way I can reference those individual log messages directly?
It seems like a custom chunker might be a way to do this?
For example, maybe the chunker could chunk at each log message and execute other per-message logic at that point?
Or is that overloading the purpose of chunkers?
I work on IPLD which provides the data structures beneath IPFS, and I’m also working on UnixFSv2 which is the next version of the file and directory data structures for IPFS.
There’s a few approaches I would recommend to solve the problem you describe and which one you want to go with depends on a few things that aren’t clear from the original post so I’ll try to describe each approach and its tradeoffs.
First of all, I wouldn’t recommend using a different chunker. IPFS does not have great APIs for addressing and working with individual chunks of a file so I’m not sure you’d get what you want out of it and you’d just end up pulling everything out of the block interface which probably excludes the benefits of many of features you chose IPFS for to begin with.
1) If all you really want is to have a webpage with a URL that points at a file, the easiest way to do this is just to refer to a normal file by line number and put some code on top of IPFS. This will also use the least storage.
If you really need a hash based immutable link to a specific log item, you have a few choices.
2) create an individual file in IPFS per log item. this might seem like overkill but it’s not too different from your chunker approach. you’d have an extra link in the graph from the file metadata to the raw data but what you’d get in return is all of the IPFS API for accessing each one as a file, and you’d be able to treat the directory structure as a nice index heirarchy.
This will increase the storage requirements though. How much depends on the size of the log lines and the size of the hash function. If your lines are roughly double the size of your hash function then you’d need about 50% more storage space.
Side note: in UnixFSv2 (in development, not ready for use) you would be able to do this without requiring another link in the graph because we have support for inlining small amounts of binary data directly into the file metadata, so the storage requirements would end up being a bit less.
3) if your log items are actually structured data, like something you would turn into JSON after pulling out rather than leaving as a string, or if the original data is already structured and being flattened to a log line, you might want to consider using IPLD directly.
This approach is more work and the tools are less developed (IPLD is a younger project than IPFS). But, you’d end up with structured data in and out, and you could still use IPFS as a data store by using either the DAG API or the Block API.
How you would accomplish this depends a lot on the language you’re using, and would require a significant time investment (you’re literally designing a new custom data structure). I’m skeptical this will be the right choice for you use case, but I wanted to put it out there as an option in case it is. If you want to explore this check out the project and log an issue with some more details and we can help you through it, but again, this is definitely more work ;) https://github.com/ipld
It sounds like what you are trying to do is have some semantics to access part of some data set. What you are running into is that UnixFS (the encoding IPFS uses by default for files) doesn't know anything about your particular semantics.
While you could decide to use a custom chunker and then address the blocks that make up the UnixFS object, it would probably be easier for you to just make a new IPLD object MyLog which is a list of log entries. You would then be able to query the individual log entries.
The underlying approach here is basically to create a new data structure for storing your data. Whether you choose to make that data structure UnixFS compatible or not is up to you.
