Github GraphQL API does not return the entire response of json file

Viewed 31

My application is a documentation website. And the documentation is stored on github. I am using the github graphQL API to copy the files from the content repository to the site repository in the build. Because then everything is local and I don't have to keep making requests.

What I did is working in parts. My problem is that some of the files are being returned incomplete. Simple example to undestant:

I have:

{
 text: "est simplement du faux texte employé dans la composition et la mise en page avant 
   impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les années 
   1500, quand un imprimeur anonyme assembla ensemble des morceaux de texte pour réaliser un 
   livre spécimen de polices de texte. Il n'a pas fait que survivre cinq siècles, mais s'est 
   aussi adapté à la bureautique informatique, sans que son contenu n'en soit modifié"
}

And the request return me(even though the text is cut off the final key appears haha):

{
 text: "est simplement du faux texte employé dans la composition et la mise en page avant 
   impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les années 
   1500, quand un imprimeur anonyme assembla
}

I don't know if it could be a file size limit. The very large files that came incomplete. The one I tested that was broken was a 16855 line json. Please someone help me :(

If I have forgotten any important detail, just let me know and I will correct it.

Now, the code:

import * as fs from 'fs'
import * as path from 'path'

const dir = path.join('public', 'teste')

const openSource = {
  githubConvertedToken: 'xxxx',
}

import fetch from 'node-fetch'

const body = {
  query: `
    query {
      repository(owner: "xxxx", name: "xxxx") {
        object(expression: "HEAD:") {
          ... on Tree {
            entries {
              name
              object {
                ... on Blob {
                  text
                }
              }
            }
          }
        }
      }
    }`,
}

const baseUrl = 'https://api.github.com/graphql'

const headers = {
  'Content-Type': 'application/json',
  Authorization: 'bearer ' + openSource.githubConvertedToken,
}

const assertDirExists = (dir_path) => {
  if (!fs.existsSync(dir_path)) {
    fs.mkdirSync(dir_path)
  }
}

fetch(baseUrl, {
  method: 'POST',
  headers: headers,
  body: JSON.stringify(body),
})
  .then((res) => res.json())
  .then((res) => {
    let content = res
    assertDirExists(dir)
    const pathdir = path.join(dir, 'aaa')
    let count = 0
    content.data.repository.object.entries.forEach((element) => {
      if (element.name.includes('json')) {
        try {
          const test = JSON.parse(element.object.text)
          fs.writeFileSync(`${pathdir}${count}.json`, JSON.stringify(test))
          count = count + 1
        } catch (err) {
          console.log('error', err)
        }
      }
    })
  })
  .catch((error) => console.log('erroooooooooo', error))
0 Answers
Related