Program need to return array data structure elements back to the program where it's called t

There are probably dozens of ways to handle this. A few ideas:

1. Using a temp file. This is the kind of thing the QTEMP library is good at. You can create and fill files with data knowing that they'll be deleted when the job ends, or before if you explicitly delete them yourself. They can act like a variable-length array, something that RPG does not currently support well. Your concurrency concern can probably be overcome if you try. RPG F-specs and non-dynamic SQL do not have a good way to choose a file by passing its name in a string, but clever use of the OVRDBF and DLTOVR commands can provide the fancy footwork for the master program to create a target file with a unique name, override it to an expected name, and call the second program. This lets the second program to be written with a common F-spec or static SQL name, but it will really write to the overridden file.

2. Pass a large array when calling your program. If you allocate and deallocate the space yourself, you can handle any concurrency concerns by ensuring that each called program gets its own array to write to. Some people might have concerns that this is wasteful of memory or inefficient. It is true that you will be allocating big chunks of memory that the called programs may not use, but is that such a problem on this operating system? Consider some things: parameters are usually passed by reference, not by value, so the contents of the array do not have to be blitted from one copy to another; RPG does not initialize data structures by setting them to blanks and zeros unless you specify the INZ keyword, so taking out space for a large array is just reserving space, not writing to it, and you can pass max used size as a parameter also to keep yourself from reading junk; allocating memory (an array or a userspace or maybe an IFS temp file) should all be about equally fast given the IBM i concept of a single-level store (a temp DB file probably has a little more overhead because the DB2 engine needs to create it and keep track of its fields and records). In other words, passing big chunks of memory to your subprograms might increase your total memory footprint, but it won't necessarily make your programs slow if you are smart about it.

3. Create a userspace object and use that as a shared memory area. I don't do this a lot because it feels more like a C technique using pointers, but it can certainly be a useful way to provide free-form space to write whatever you want. The system API's often use these userspaces to pass information out of a called program. An example of using pointers in RPG to read a userspace can be found here. You can handle concurrency by creating separate, uniquely named userspaces in QTEMP, of different sizes if you want, and pass their name and size back to the calling program for it to read. Their variable size can keep your memory footprint smaller than fixed-size arrays. If IBM chose to use them in API's as extensively as they did, it must be a good technique. You can keep the pointer code contained in one or two small sections and even clean it up by mapping a data structure onto the userspace for easier reading and writing using the BASED keyword.

4. You can create a callback program that delivers one data structure entry every time it is called. This may be the lowest memory method, depending on where your data is being loaded or created from. It is similar to the lazy loading used in other systems under the name of iterators or enumerators. The idea is to call a function a number of times and to get one entry each time. At no point does the whole thing need to be loaded into memory at once. The same example given above shows one way to do this technique if you look at the procedure pointer section of the code rather than the userspace section. Another way that I have tried this is to make a prototype of a data structure with three procedure pointers for the functions Current, MoveNext, and Reset. RPG makes you do more of the plumbing yourself than an IEnumerable in Microsoft.NET, but the concepts work for lazy-loading any size dataset if you are willing to write it yourself. You won't get compile-time checking, but it is not that hard to write in this style correctly once you understand the pattern. It reminds me of simulating object-oriented programming in C: set yourself some rules to follow and of course it works.

5. Another callback based example can be seen in how the DB2 SQL system interacts with External User Defined Table Functions (UDTF's) written in RPG or C. An example of this can be found in Birgitta Hauser's excellent article The Power of User-Defined Table Functions. I am not suggesting that you actually create a UDTF for your purpose (although that would be one legitimate, if higher overhead, solution), but instead to look at the callback mechanism that they use when calling an external program. It calls the subprogram multiple times: open (to initialize if necessary), an unrestricted number of times for each fetch until it is done, and close (to clean up and close files if necessary). Using a pattern like this is another way to pass only one data structure at a time rather than a whole array, and may be very attractive if your potential data sizes are large or unpredictable.

Hope this helps. There are probably other ways to do this that I'm not thinking of right now. Maybe someone else will chime in with them.