Journal TOC generation does no add the title

[svenXY]

Prerequisites

• I am using either 0.7 stable with the 0.0.12 tag or the latest compiled Neovim version
• I am using the latest version of the plugin

Neovim Version

NVIM v0.9.0-dev-2154+g89722ddfa-dirty

Neorg setup

require('neorg').setup {
    load = {
        ["core.defaults"] = {},
        ["core.norg.journal"] = {
          config = {
            toc_format = function(entries)
              local months_text = {
                "January",
                "February",
                "March",
                "April",
                "May",
                "June",
                "July",
                "August",
                "September",
                "October",
                "November",
                "December",
              }
              -- Convert the entries into a certain format to be written
              local output = {}
              local current_year
              local current_month
              for _, entry in ipairs(entries) do
                  -- Don't print the year and month if they haven't changed
                  if not current_year or current_year < entry[1] then
                      current_year = entry[1]
                      table.insert(output, "* " .. current_year)
                  end
                  if not current_month or current_month < entry[2] then
                      current_month = entry[2]
                      table.insert(output, "** " .. months_text[current_month])
                  end

                  -- Prints the file link
                  print(vim.inspect(entry))
                  table.insert(output, entry[4] .. string.format("[%s]", entry[5]))
              end

              return output
          end,
      },
    },
        --[[ ["core.keybinds"] = {}, ]]
        ["core.norg.concealer"] = {
          config = {
            icon_preset = "diamond",
          },
        }, -- Allows for use of icons
        ["core.norg.dirman"] = { -- Manage your directories with Neorg
            config = {
                workspaces = {
                    private = "~/neorg/private",
                },
                default_workspace = "private",
                autodetect = true,
                autochdir = true,
            }
        },
        ["core.norg.completion"] = {
            config = {
                engine = "nvim-cmp"
            }
        }
    }
}

Actual behavior

I just added

@document.meta
  title: foobarbaz
  description:
  authors:
  categories:
  created:
  version:
@end

at the top of a journal entry, but it did not add the title but kept adding the day as the title.

The entry is always:
{ 2023, 3, 17, "{:$/journal/2023/03/17:}", "17" }

Expected behavior

The entry should then be:
{ 2023, 3, 17, "{:$/journal/2023/03/17:}", "foobarbaz" }

as then create_toc() would properly set the document title as the link name.

Steps to reproduce

:Neorg journal today

add a @document.meta tag to the journal entry

:Neorg journal toc update

Potentially conflicting plugins

No response

Other information

The documentation does not mention that adding the document.meta is required.

Furthermore, it would be great if there could be some means as to read the title from the first line (or first heading) of the file.

Help

Well, I'm new to lua and would need a lot of help, but willing to help if I can

Implementation help

No response

[danymat]

Hello,
I tried reproducing the issue and I don't get it on my end. Indeed, it is working correctly. Can you confirm that the issue still persists on your side ?

The documentation does not mention that adding the document.meta is required.

It is not required. By default, it's the name of the file that is taken.

[svenXY]

Hi, thanks for taking this up. Yes, the problem persists. I just tried again and instead of using @document.meta, it always uses the filename (here: the day). And looking at the entries (vim.inspect(..), I can see that toc_format already receives the entries without the metadata. Maybe I need to configure an additional module or install something?

[svenXY]

Maybe it's a treesitter problem.

For the meta tag above, I get

ranged_verbatim_tag [0, 0] - [8, 4]
  name: tag_name [0, 1] - [0, 14]
    word [0, 1] - [0, 9]
    word [0, 10] - [0, 14]
  content: ranged_verbatim_tag_content [1, 0] - [8, 0]
    pair [1, 0] - [2, 0]
      key [1, 0] - [1, 5]
      value [1, 7] - [1, 16]
    pair [2, 0] - [4, 0]
      key [2, 0] - [2, 11]
      value [3, 0] - [3, 24]
    pair [4, 0] - [6, 0]
      key [4, 0] - [4, 10]
      value [5, 0] - [5, 19]
    pair [6, 0] - [7, 0]
      key [6, 0] - [6, 7]
      value [6, 9] - [6, 19]
    pair [7, 0] - [8, 0]
      key [7, 0] - [7, 7]
      value [7, 9] - [7, 14]
  ranged_verbatim_tag_end [8, 0] - [8, 4]

and in the documentation, I find this without any "verbatim"

update: I just checked and treestiier claims to have successfully installed the norg_meta parser

I think the problem must somehow be with the get_title() function.

I'm not too experienced with lua/neovim development, but - how can I run something like module.required["core.integrations.treesitter"].get_document_metadata(0)as a vim command to see what it returns?`

[danymat]

I'm not too experienced with lua/neovim development, but - how can I run something like module.required["core.integrations.treesitter"].get_document_metadata(0)as a vim command to see what it returns?`

Hello, you can use the get_module api available in neorg:

:lua vim.pretty_print(neorg.modules.get_module("core.integrations.treesitter").get_document_metadata(0))

I'm still not able to reproduce your issue.. Make sure you're on latest neorg version and latest parsers.

[svenXY]

update:

{
  categories = "created: 2023-03-17",
  description = "authors: svenXY",
  title = "dfggfhghdghdg",
  updated = "2023-03-29",
  version = "1.0.0"
}

was returned. Despite the fact that empty fields seem to break this somehow, the title is filled correctly.

:lua vim.pretty_print(neorg.modules.get_module("core.integrations.treesitter").get_document_metadata(0).title)correctly returns the title.

Now I'm out of ideas...

[svenXY]

Could this be an issue?

[nvim-treesitter] [3/7] Extracting tree-sitter-norg...
[nvim-treesitter] [3/7] Creating temporary directory
nvim-treesitter[norg]: Could not create tree-sitter-norg-tmp
mkdir: tree-sitter-norg-tmp: File exists
nvim-treesitter[norg]: Failed to execute the following command:
{
  cmd = "mv",
  opts = {
    args = { "tree-sitter-norg-tmp/tree-sitter-norg-6348056b999f06c2c7f43bb0a5aa7cfde5302712", "tree-sitter-norg" },
    cwd = "/Users/me.local/share/nvim",
    stdio = {
      [2] = <userdata 1>,
      [3] = <userdata 2>
    }
  }
}
mv: rename tree-sitter-norg-tmp/tree-sitter-norg-6348056b999f06c2c7f43bb0a5aa7cfde5302712 to tree-sitter-norg: No such file or directory
[

[danymat]

Oh yes maybe, I recommend deleting this temp folder + the so files, and run :Neorg sync-parsers again

To search for the so files:

find ~/.local/share/nvim -name "*norg*.so"
# /Users/danymat/.local/share/nvim/lazy/nvim-treesitter/parser/norg_meta.so
# /Users/danymat/.local/share/nvim/lazy/nvim-treesitter/parser/norg.so

[svenXY]

[nvim-treesitter] [1/3] Treesitter parser for norg_meta has been installed
[nvim-treesitter] [1/3] Creating temporary directory
[nvim-treesitter] [1/3] Extracting tree-sitter-norg...
[nvim-treesitter] [1/3] Compiling...
[nvim-treesitter] [2/3] Treesitter parser for norg has been installed
nvim-treesitter[norg]: Error during compilation
cc1plus: fatal error: src/scanner.cc: No such file or directory
compilation terminated.

[svenXY]

I'm on a MacBook here with the suggested settings for treesitter:

require("nvim-treesitter.install").compilers = { "gcc-12" }

[danymat]

Could you print the output of :checkhealth nvim-treesitter until OS Info included ?

[svenXY]

Interestingly enough, it worked once, i.e. the title for the two nodes with metadata was added correctly once, but a subsequent repetition of the same command again only returned the filename.

[danymat]

@vhyrro maybe totally off, but isn't this a consequence of vim.schedule not working properly ?

[svenXY]

==============================================================================
nvim-treesitter: require("nvim-treesitter.health").check()

Installation ~

• OK tree-sitter found 0.20.7 (parser generator, only needed for :TSInstallFromGrammar)
• OK node found v19.8.1 (only needed for :TSInstallFromGrammar)
• OK git executable found.
• OK gcc-12 executable found. Selected from { "gcc-12" }
  Version: gcc-12 (Homebrew GCC 12.2.0) 12.2.0
• OK Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

OS Info:
{
machine = "arm64",
release = "22.4.0",
sysname = "Darwin",
version = "Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000"
} ~

Parser/Features H L F I J
[...]

• norg . . . . .

[svenXY]

checked again after running sync-parsers again, which - this time - did not error out.

Parser/Features H L F I J
  - norg                ✓ . ✓ . ✓
  - norg_meta           ✓ . . ✓ .

but the toc has not changed

[svenXY]

It works the first time I enter the toc buffer, i.e. not having it open, then Neorg journal toc update works, then - while having it open, running the command again, it fails (filename instead of title), closing the toc buffer (:bw), then running the command again - works again.

I thought it might be due to treesitter lazy loading, but removing event = "BufRead", from my treesitter lazy config did not make it better.

@danymat maybe you can reproduce, but only ever ran it once?

[danymat]

Hello @svenXY, seems like it's still working on my end (cannot reproduce). The only thing left for me to debug is to provide me a minimal config so that I can reproduce on my end

[danymat]

@svenXY, by the way, are you on latest neorg ?

[quentin-fox]

I've experienced the same issue on my setup, it's a major frustration as without the title, the TOC linking doesn't seem to work, going instead to a 404 buffer.

Digging into it a little bit, running the following line when I'm in one of my .norg journal files returns the document.meta key/value pairs correctly:

print(vim.inspect(neorg.modules.get_module("core.integrations.treesitter").get_document_metadata(0)))

After opening the index.norg, I can also run the following commands without any issues:

print(vim.fn.bufadd('2023-04-03.norg'))
print(vim.inspect(neorg.modules.get_module("core.integrations.treesitter").get_document_metadata(bufnr)))

Where the file name is a file that is in the same directory as the index.norg, and the bufnr passed to get_document_metadata is the value printed out by the first function.

However, running :Neorg journal toc update does not populate the "title".

If I run :ls!, I can see that all of my journal files were loaded into unlisted buffers, so it's not that that's failing. They appear in this list the same way that they appear when I manually do vim.fn.bufadd('2023-04-03.norg'), which then works.

Adding a print(vim.inspect(meta)) here:

neorg/lua/neorg/modules/core/norg/journal/module.lua

Line 228 in 01d8a9b

local meta = module.required["core.integrations.treesitter"].get_document_metadata(buffer)

Each of the buffers returns the following table as the meta:

{ "title" = "" }

Maybe we need to schedule the bufadd and the treesitter query in separate Lua schedule functions? It's almost like tree sitter needs an extra tick to process the file that it doesn't get when we add the buffer and try to parse it in the same Lua function.

No idea if that mental model is correct, but will try to replicate soon. This is a real blocker for me adopting neorg for my journaling/task tracking.

[quentin-fox]

I was able to accurately recreate the issue using this repo (linked below) - rather than pulling the title from the document.meta, it just populated the day of the month that the file was created (e.g. [8])

https://github.com/quentin-fox/neorg-issue

You should be able to run setup.sh and have it not interfere at all with your existing neovim config/installations, since I set it to use a brand new temp directory as the XDG_(CONFIG/DATA/STATE)_HOME.

Lmk if there are any issues with it!

The first time that the script loads up nvim, it'll have a bunch of issues, but the second time it should load up fine.