How to maintain an emacs-based knowledge base?

I think this excellent document on how to use org-mode to it's fullest potential will be very helpful to you: "Org Mode: Organize Your Life In Plain Text". It is lengthy reading, but trust me, completely worth the effort.

UPDATE: You can use the remember-mode section mentioned in the document for your use-case. (I use it for the same use-case) Remember-mode is extremely handy to make quick notes. I use it when I have to store random observations or information that won't go anywhere else. I use the following templates for remember:

(setq org-default-notes-file (concat org-directory "/remember-notes.org"))
(setq org-remember-templates
  `(("Todo"    ?t "* TODO %?\n  %i\n" ,(concat org-directory "/remember-notes.org") bottom)
    ("Misc"    ?m "* %?\n  %i\n"      ,(concat org-directory "/Notes.org")   "Misc")
    ("iNfo"    ?n "* %?\n  %i\n"      ,(concat org-directory "/Notes.org")   "Information")
    ("Idea"    ?i "* %?\n  %i\n"      ,(concat org-directory "/Notes.org")   "Ideas")
    ("Journal" ?j "* %T %?\n\n  %i\n" ,(concat org-directory "/journal.org") bottom)
    ("Blog"    ?b "* %T %? :BLOG:\n\n  %i\n" ,(concat org-directory "/journal.org") bottom)
    ))

As you can see, misc notes and other information goes in the notes.org file under the headings Misc and Information. If the note I'm making doesn't fall in any of the categories defined above, it gets filed in the default file (remember-notes.org) and I can always refile it to another location at a convenient time. This makes my note-taking, jotting down random ideas, and such things extremely simple, without taking away the focus from the job I'm currently doing.

I personally keep a list of project directories with somewhat similar structure. Each has a tasklist.org, a tracking subdirectory (where I do project estimates and time tracking and always maintain a diary which is the main thing for the project - it will have links to other files for the project), a docs subdirectory which usually consist of stuff I'm going to publish (docs for the project, proposals etc.). I get my agenda-files to the tasklist.org in each of the subdirectories so that my agenda works fine.

I think the organisation of the data would change a little in your case (perhaps topics like "functional programming" etc.). I'm sceptical of how much a hierarchical structure would help since that would confine you into one way of looking at things (tags vs. folders again). Here are some things that come to mind.

1. Keep a 'master' org file that has links to all the 'interesting' top level pieces of the other content (similar to the diary I mentioned above).
2. Tag all your material properly (you'll settle on a set of useful tags after a while) and then use the Tag search feature to search through the files quickly. This assumes that all the files are in your agenda-files though.
3. Finally, if your data is too exotic to put into a structure, you can consider using a full text indexer (like xapian) and integrate that into your Emacs. There was some discussion of this over here.

Update : 26/Nov/2019

I recently came across the hyperbole package which claims to be a information organiser. I haven't used it but I'm quite tempted to and will update this when I do.

[org-mode] is great for managing somewhat 'actionable' items, but I keep adding things of a more general nature, that I won't be needing on a day-to-day basis (how-tos, reading notes etc), so it's getting slow and hard to manage.

I'm a follower of David Allen and his Getting Things Done methodology. I'm using Emacs for three of the lists he recommends:

• Next Actions

• Project Resources

• The Someday/Maybe list

The material I'm concerning myself with doesn't fit the /projects/tasks/sub-tasks paradigms, they are more like little knowledge nuggets on selected topics, which are inherently more complex to classify and manage.

I've been wondering what kind of structure could be used to handle that kind of information (classification and retrieval), and if there are maybe other modes that could help with the job ?

For this kind of information I've migrated away from emacs. Instead I keep a directory ~/etc/howto, and in that directory I put files that contain "little knowledge nuggets on selected topics", where the key criterion is that the information has long-term value.

I could search this directory with Emacs, but my Emacs Lisp is not so hot, so I wrote a howto shell script instead (some error checking omitted for clarity):

case $# in
  1) ;;
  *) echo "Usage: $0 <topic>" 1>&2; exit 2 ;;
esac

topic="$1"

# Note the ordering: first exact matches, then beginning matches, then any matches

set xxx `find $HOME/etc/howto/. -name "$topic"     -not -type d -print` \
        `find $HOME/etc/howto/. -name "${topic}?*" -not -type d -not -name '*~' -print` \
        `find $HOME/etc/howto/. -name "?*$topic*"  -not -type d -not -name '*~' -print`
shift

case $# in
  0) echo "No file found matching *$topic*" 1>&2 ; exit 1 ;;
  *) for i
     do
       less "$i"
     done
     ;;
esac

Examples include:

• howto football brings up three nuggets, in this order:

  • Instructions to give to my wife on how to record a football game on the computer

  • Instructions for me as exactly what to take and how to dress when I have tickets to a football game

  • Instructions for transcoding a football game so it can be transmitted over the net and viewed away from home

• howto filesystem brings up instructions on how to copy a filesystem

• howto batteries brings up a list of recommended rechargeable batteries

One reason I don't use Emacs is that my real script is a little more complicated than what you see above: it also handles PDF and djvu files, so for example howto razor brings up a djvu document of the manual which came with my electric razor.

I have over 500 items in the main directory or in subdirectories, and even at this scale the system works quite well for me. I hope you may find it helpful too.

I have tried several ways to manage a knowledge base in the past. I have a bunch of "knowledge nuggets" (btw, thanks, I like that term a lot) on all sorts of diverse topics ranging from how to set up apache tomcat ssl cert, to checklists for doing a monthly family budget, to keeping a list of weights and reps completed on workouts.

I've tried keeping these in a wordpress blog, on a personal wiki, using pen and paper, etc.

In the end, emacs and org-mode is the clear winner for me. I love to have the ability to start simple, and build more complex functionality as I need it. I've used a lot of the tips described by Sacha Chua.

In my case, I always end up with a bunch of notes (more formal and organized) mixed in with action items (less formal). In general I maintain one master "action item" list and then create a separate file for notes on each topic. So far, grep has worked well for me to quickly find the file containing notes. I'll often create a emacs bookmark C-x r m to quickly navigate to notes files as well.

Simple Blogs, CMS, and wikis (like Drupal and Wordpress) are good at classification and retrieval. Maybe you could export org files to html and publish them to a blog, cms or wiki? It might not be too difficult to hook into the blog/wiki/cms tagging capability.

At work, we use a wiki (actually, several - a global wiki plus a wiki per project) for this. It's ideal for non-hierarchical data, but can also be used for hierarchical data. It's formattable, hypertextual, linkable-to, searchable, shareable but also ownable, it maintains history, and other good stuff.

Personally, i used to also use a wiki for this. But these days, i generally just forget things instead. Far easier.

You can speed up org-mode by keeping your notes in a separate file (or one for each broad topic) which is not included in the usual list of agenda files. Customise org-agenda-files to see the list.

Use org-remember to enter notes quickly without interrupting your flow. Either tag them at the time, or save them somewhere to be refiled later. You can use a tag in the remember template (customise org-remember-templates) to mark notes for refiling, and use a custom agenda search (org-agenda-custom-commands) to list them.

Tag each note with relevant topics, and use the agenda view's search facilities to find them. You can define a custom search which knows to look in the right files, or you can visit the file and restrict the agenda search to just that file.

I keep a file of notes, and your question has just inspired me to go back and start tagging them all. Works a treat!