TDD - Approach for a file repository system

J Slick 0
Viewed 33

I would like suggestions on how TDD would approach this problem. Sorry if it seems too long, but this was a real task and there was much more which I did not include. I suppose the obvious thing is to call this an epic, and reduce it to stories, but I don't think it could be so simple. During development, I created several test functions, and manually ran them consistently with development. My solution has been in production six months and has done the Right Thing on over 400K documents.

Design and code a file management system such that:

  1. The solution iteratively receives a single PDF and parameters including: {file category [A, B], file subcategory [S1, S2, S3, S4], file version (one or two chars)}. Example: file123.pdf, A, S4, X.

  2. Copy each PDF to a repository. The path to the repository root must be retrieved from a config database. If the repository root does not exist or cannot be reached, the solution will not be able to create it, nor be able to copy any files.

  3. If the repository does exist and can be reached, copy each file to a destination somewhere below the repository root. Grow the repo dynamically, organizing the file destination by file category, subcategory and one or more sequentially numbered subdirectories containing PDF's. Example: A/S1/1, A/S1/2, A/S1/3, etc.

  4. Each PDF subdirectory must contain a single index file containing one unique line for each PDF.
    Index line format is: {filename prefix, underscore, subcategory, underscore, version, .pdf}.

    Example: file123_S4_X.pdf.

    File category [A, B] is not part of the file name nor of the index entry.

  5. Each PDF directory may not exceed a maximum aggregate size, defined in a configuration database. The aggregate size must account for all PDF's in the directory plus the size of the index file.

tdd
2 Answers

I would like suggestions on how TDD would approach this problem.

You probably end up with slightly different answers depending on whether you are considering the problem of building a system like this from scratch, or trying to build a working version of a system under control to reduce the risk of future changes.

The basic idea is that we want to be able to filter the modules of the implementation into two basic categories.

  • modules that are easy to test (but can be arbitrarily complicated)
  • modules that are "so simple there are obviously no deficiencies" (but can be difficult/expensive to test).

The Red/Green/Refactor loop works well for the first category; the second category often requires different strategies (human inspection, interactive testing, etc).

The design is normally built so that we can provide substitute implementations of the "simple" modules, so that we can more easily control their behavior when testing the interaction with other modules in the system.

If you like: one manifestation of the fact that the design is "test driven" is that complicated modules have the affordances we need to isolate them from the parts of the system that are hard to test.

So an abstraction like a "repository client" will typically have a small dumb piece that actually talks to the repository, an isolated clever piece that figures out the messages to send and how to interpret the responses, and a bit of glue. Isolated clever piece gets "unit tests"; the small dumb piece gets inspection, and the glue you typically test using the "real" clever implementation and a substitute for the small dumb piece".

I suppose the obvious thing is to call this an epic, and reduce it to stories, but I don't think it could be so simple.

Greenfield, that can be pretty close. What often happens is that we build a much simpler program than the one we need, refactor it until we have a library with a big collection of re-usable parts, and then build a more complicated program using the parts available in the well tested library.

As an example: specifying example behaviors for a random number generator is challenging, especially before it becomes clear what abstractions we want to use to exchange information with it. But if we write a program with a deterministic behavior, we can then refactor that design, experimenting with different abstractions until we find one that makes sense, and then work out how to modify the design so that we can substitute non-deterministic behaviors for deterministic ones (having already discovered how the calling code wants to interface with it).

VoiceOfUnreason0

There are many ways one could approach a problem complex like this one. TDD isn't a design methodology, so you'd should strongly consider sketching out a plan before you start coding. Not too elaborate a plan, because, as Helmuth von Moltke the Elder wrote, no plan survives contact with the enemy. Thus, plan just enough that you have a strategic understanding of the problem, but be ready to throw away your plans when tests tell you that you were wrong (they will).

You can do TDD in outside-in style or a bottom-up style. With whole systems like the one outlined in the OP, by default I'd lean towards outside-in in the spirit of GOOS. Start with some high-level tests that establish that a system even exists and can take input. Then elaborate on the details of those interactions by dropping from high-level tests to more detailed unit tests, as needed. GOOS elaborately describe this process. I also give a similar example in Code That Fits in Your Head.

A common pitfall when presented with a problem like the OP is to get lost in the details and start creating an elaborate abstraction of the file system. This is most likely not a good idea. You probably don't need to model all details of a file system. While you need to move files and perhaps read the contents of a directory, do you need to be able to set ACLs on files? Model reparse points? Set archive or read-only bits?

You're probably better off keeping the Dependency Inversion Principle in mind: Abstractions do not depend on details; details depend on abstractions. In a context like this, it implies that you shouldn't try to model a file system but rather the actions your system has to perform.

Still, at a lower level you may need to be able to model and unit test something that looks like a file system. Most object-oriented programmers immediately reach for a set of interfaces like IDirectory and IFile, which you can do. Consider alternatives, however. A file system is just a tree, so why not model it as one?

Mark Seemann0
Related