Libre Bib Manual


Install and Setup


O.S. Requirments

| OS Version         | Package | Tested      | Notes       |
|--------------------+---------+-------------+-------------|
| mx linux 21.x      | native  | yes         |             |
| mx linux 21.x      | generic | in-progress |             |
| Ubuntu 18.04       | native  | in-progress |             |
| Ubuntu 18.04       | generic | in-progress |             |
| Ubuntu 20+         | generic | no          | manual deps |
| Debian ??          | generic | no          | manual deps |
| RedHat ??          | generic | no          | manual deps |
| Windows ??, CygWin | generic | no          | manual deps |
| MacOS ??, brew     | native  | no          | manual deps |
| MacOS ??, brew     | generic | no          | manual deps |

Install Package

Required Packages

Optional Packages

Required Packages for Building

Config


Setup libre-bib project

Run:

bib setup-bib

Fix any errors then run it again, until no more errors.

If you are planing on using a remote DB, then see the "Configure ssh" section.


Configure the DB

Install and test

The DB packages, mariadb-client and mariadb-server, have been installed on the remote server (or local sever if you are doing this all on one server). Most likely the mariadbd process will already be running. Verify this with:

ps -fC mariadbd

If you don't see it running, you'll need to consult the mariadb docs to get it running.

The first one is a good source for quickly getting going. Depending on your distribution, you may need to do things a bit differently.

Test the connection on the server system

sudo mysql -P 3306 -u root -p

Most likely you'll use your sudo password, or the password you setup for the mysql DB root user.

Create DB, Users, and Grants

While signed in as root user to the DB type these commands. Replace the $cgNAME variables with the values of those variables in your project/conf.env file. You can change those now or just use the example names to try things out.

sudo -s
mysql -P 3306 -u root -p
create database $cgDbName;
show databases;
create user 'admin'@'localhost' identified by 'ADMIN-PASS';
grant all privileges on *.* to 'admin'@localhost;

create user '$cgDbUser'@'localhost' identified by 'USER-PASS';
grant all privileges on $cgDbName.* to '$cgDbUser'@localhost;

flush privileges;

select user from mysql.user;
show grants for 'root'@localhost;
show grants for 'admin'@localhost;
show grants for '$cgDbUser'@localhost;

quit;

If that doesn't work look at the cgDsn variable setting in project/conf.env. It should be set to $cgLocalDsn for local access. Try again, If that works, your conf.env setting are good for continuing (skip the ssh section and other areas mentioning remote db access).


Configure ssh

In your ~/.ssh/ dir you should see a libre-bib.ssh file. For this to be setup properly edit your project/conf.env file. Set the variables:

| conf.env var   | Description                              |
|----------------+------------------------------------------|
| cgDbHost       | keep this set to the localhost IP        |
| cgDbName       | name of the mysql database               |
| cgDbPortRemote | remote port, on project's system.        |
| cgDbLocalPort  | port for mysql on the remote system      |
| cgDbUser       | DB user with grants to cgDbName          |
| cgDbPassHint   | hint for the password prompt             |
| cgDbSshUser    | user that can login to the remote system |
| cgDbSshKey     | key login to the remote system           |

Remove ~/ssh/libre-bib.ssh file and run again:

bib setup-bib

If the ~/ssh/libre-bib.ssh file looks OK, add following line top of your ~/.ssh/config file (or near a Host config for your system).

Include libre-bib.ssh

If you want to add more ssh options for the Host, don't add them to libre-bib.ssh, because that could be overwritten if project/conf.env is changed. Create another Host line with the same host name and add the option you want.

Test the tunnel

Do this after you have setup the DB, and you have tested connecting locally.

In a terminal ssh to the remote system.

ssh $cgDbSshUser@$cgDbHostRemote

Leave the terminal window open and start another terminal window. In the new terminal window type:

telnet 127.0.0.1 $cgDbPortRemote

You should see "Connected to 127.0.0.1" and probably password prompt. Exit with ctrl-C or ctrl-] then "quit".

Now test the connection to the database:

mysql -P $cgDbPortRemote -u $cgDbUser -p -h 127.0.0.1 $cgDbName

If that doesn't work, look at the error message and see what needs to be fixed. Check: db user name, db name, ports, grants and other settings on the db system.

If that does work, try connecting with the "bib" command.

bib connect

If that doesn't work look at the cgDsn variable setting in project/conf.env. It should be set to $cgRemoteDsn for remote access. Try again, If that works, your conf.env setting are good for continuing.


Using libre-bib


Quick Start

This shows a minimal setup with a local DB. This will use the example files and the minimal default configuration.

For a detailed example see Appendix "A Full Example."

mkdir -p project/biblio
cd project/biblio
bib setup-bib      # This creates your default conf.env file
edit conf.env      # Uncomment and set these values
    cgDbName="YOUR-DB-NAME"
    cgDbUser="YOUR-DB-USER"
    cgDbPassHint="YOUR-HINT"
bib setup-bib      # Your project are will be setup
bib                # List the bib commands
bib connect        # Connect to DB to cache the  password
bib import-lo      # Import the biblio.txt file
bib ref-new        # Use DB values for any new REFs
bib ref-update     # Update REFs with any DB changes
libreoffice example.odt
    # Load styles from another odt file (optional)
    # Add bibliography to end of document
    # Set the bibliography Type > Brackets to "none"
bib style-update   # Define the bibliography > Entries for the different Types
bib style-save     # Run this if you change bibliography > Entries

libre-bib Tour

Files and Dirs

This is a quick summary of the directories and files setup in your project directory. The details will be describe in later sections as they are used.

The bib commands will notice changes and rebuild any dependent files as needed. So you might see more things running than what you've seen before. The "Env-Var" column shows the variable for the File-Dir. The Cmd column shows the command or commands that create or use the File-Dir.

$ cd $HOME
| File or Dir                | Var / Cmd                                |
|----------------------------+------------------------------------------|
| conv.env                   | Cmd: setuup-bib                          |
| biblio.txt                 | Var: $cgLoFile;  Cmd: setup-bib          |
| biblio-note.txt            | Var: $cgLoFile;  Cmd: setup-bib          |
| key.txt                    | Cmd: setup-bib                           |
| example.odt                | Var: $cgDocFile; Cmd: setup-bib          |
| librarything.tsv           | Var: $cgLibFile; Cmd: setup-bib          |
| status/                    | Var: $cgDirStatus; Cmd: setup-bib        |
| .... import-lo.date        | Cmd: import-lo                           |
| .... backup-lo.date        | Cmd: backup-lo                           |
| .... import-lib.date       | Cmd: import-lib (from lib)               |
| .... update-lo.date        | Cmd: update-lo (from lib)                |
| backup/                    | Var: $cgDirBackup;  Cmd: setup-bib       |
| .... backup-lo.csv         | Var; $cgBackupFile; Cmd: backup-lo       |
| .... backup-lo.csv.bak     | Cmd: backup-lo                           |
| .... backup-lo.csv.bak.~2~ | Cmd: backup-lo                           |
| .... backup-lo.csv.bak.~1~ | Cmd: backup-lo                           |
| tmp/                       | Var: $cgDirTmp; Cmd: setup-bib           |
| .... .pass.tmp             | Var: $cgDbPassCache; Cmd: connect        |
| .... biblio.txt            | Var: $cgDirTmp/$cgLoFile; Cmd: export-lo |

DB Tables

| Variable  | Default        | Used by               |
|-----------+----------------+-----------------------|
| $cgDbName | biblio_example | Data Base Name        |
| $cgDbLo   | lo             | import-lo,  export-lo |
| $cgDbLib  | lib            | import-lib, update-lo |
| $cgDbBib  | bib            | import-lo             |
|           | join_lib_lo    | update-lo             |

Annotated conf.env

Understanding the variables in the conf.env file will probably give you the best understanding of how the libre-bib application works.

The conf.env files are the core configuration files for the libre-bib app. They are executed in this order, so the last definition wins.

1. . /opt/libre-bib/etc/conf.env
2. . ~/.config/libre-bib/conf.env   # $cgDirConf Optional
3. . ./conf.env

Commands

Commands are entered after "bib". For example: "bib setup-bib"

Just type "bib" for a quick list of commands.

To bring up this manual in your default browser, type: "bib help"

More than one command can be used. They will be run in order. Execution will stop at the first command that fails. ("ignored" errors are OK.) For example, you have modified some entries in biblio.txt and you have modified your libreoffice document with these new references. You could run: "bib import-lo update-lo ref-new ref-update"

If a command doesn't run, it could be the files are up-to-date. If you want to force a command to run, remove the status/ file associated with the command, or "touch" one of the dependent files. For example: import-lo will only run if the biblio.txt file is newer than the last time import-lo was run. To force it to run: "rm status/import.lo.date"

Cmd: setup-bib

After editing the conf.env file, run this command to define the directories and initial files, in the current directory.

Cmd: connect

Run this to setup and verify the DB connecton is working.

Cmd: check

Run this to do a quick check that everything is correctly defined.

Cmd: edit

This will run: $EDITOR $cgDocFile &

Cmd: import-lo

Import any changes to $cgLoFile (biblio.txt) to the lo table. The lo table will be backed-up in the DB.

Cmd: export-lo

Export the lo table to $cgLoFile (biblio.txt). The existing biblo.txt file will be backed up to $cgDirBackup.

Cmd: backup-lo

Backup the lo table to a cvs file in $cgDirBackup.

Cmd: import-lib

Import the librarything.tsv file to the lib table.

Cmd: update-lo

After the lib table is created or updated, run update-lo to copy any missing fields in lo from the lib table.

Cmd: ref-new

New biblio {REF} tags have been added to your odt file. Run this command to update your odt file with the current biblio entries found in the lo table. If there are no new entries, the file will be unchanged.

If the file is changed, the original file will be found in the backup/ dir. So your odt file can be restored if there are problems.

If the lo table has been updated with different values, then run the ref-update command.

Internal: see /opt/libre-bib/etc/cite-new.xml for the template that will be used.

This will format the entries with the "Endnote Characters" style, and insert the non-empty bib-field values.

Cmd: ref-update

If the lo table has been updated with different values, then run this command to update the odt file with the new values. This command will not modify any new {REF} tags.

The original file will be found in the backup/ dir. So your odt file can be restored if there are problems. It could be there are no changes to the file, but this command doesn't check for difference, it just replaces all of the biblio-entries it finds in the odt file.

Internal: see /opt/libre-bib/etc/cite-update.xml for the template that will be used.

This will only update non-empty bib-field values. The style won't be touched.

Cmd: save-style

After setting the bibliograpy entry styles, run this command so the style is saved. That way it can be restored, if the odt file is recreated.

Cmd: update-style

Replace the bibliograpy entry styles from the styles saved with save-style.

Cmd: status

Report on the size of tables and files. Also list what needs to be updated.

Cmd: clean

Remove temporary files. This will also remove the DB password cache file.

Cmd: version

Report the current bib version. Include this in any bug reports.

Cmd: add, edit

Run: $EDITOR biblio.txt &

Cmd: help

Show the libre-bib manual in a browser window.


Appendix


A. Backups


B. Customizing the defaults

If you are managing multiple bibliographies, you might have some common settings. For example, most of the things related to a remote DB will be the same.

The user config file is the best place for defining the common settings: $cgDirConf/conf.env

If you change the /opt/libre-bib/etc/conf.env file, you will need to rebuild some things. Here are the steps:

cd /opt/libre-bib/etc
edit conf.env
bash -n conf.env   # syntax check
cd BIB-PROJECT     # any of your bib project dirs
bib rebuild        # update user default file, and conf.php

Source /opt/libre-bib/etc/conf.env and conf.env in a bash script call your own Makefile, other bash scripts, or php scripts to run things. Your php scripts could include /opt/libre-bib/etc/conf.php to define the ENV vars as globals, or just use $ENV['cgVarName'].


C. Emacs Org Mode - Outine

doc/example/example-outline.org


D. Full Example

This assumes you have everything installed and working. This will use the example files.

$ cd $HOME
$ mkdir -p project/biblio
$ cd project/biblio
$ bib
Usage:
    bib [-n] Cmd
Cmds:
    import-lo, export-lo, backup-lo, restore-lo
    import-lib, update-lo
    ref-new, ref-update
    status, setup-bib, clean, connect, version, help
$ bib help
Error: Missing conf.env, copying it now
Edit conf.env with your details. Uncomment the ones you are changing.
Then run: bib setup-bib
Usage:
        bib [-n] Cmd
Cmds:
    import-lo, export-lo, backup-lo, restore-lo
    import-lib, update-lo
    ref-new, ref-update
    status, setup-bib, clean, connect, version, help
$ ls
conf.env*

If you accidentally ran bib in a directory that is no going to be a bibliography directory, just delete the conf.env file.

$ emacs conf.env
change:
    export cgDbHostRemote="NAME.example.com"
    export cgDbPassHint="b4n"
    export cgDbUser="$USER"
    export cgUseRemote=false
    export cgSshKey="$HOME/.ssh/id.KEY-NAME"
    export cgUseLib=false
to
    export cgDbHostRemote="myserver.example.com"
    export cgDbPassHint="fav-pet"
    export cgDbUser="example"
    export cgUseRemote=true
    export cgSshKey="$HOME/.ssh/id.mysys"
    export cgUseLib=true
save, and exit

$ bib setup-bib
Missing example.odt. Copy an example from
/opt/libre-bib/doc/example/example.odt

Missing: biblio.txt. Copy an example from
/opt/libre-bib/doc/example/biblio.txt

Missing librarything.tsv. Copy an example from
/opt/libre-bib/doc/example/librarything.tsv
Manually update it with an export from Library Thing.
$ ls
backup/          biblio.txt  conf.env~*   key.txt           status/
biblio-note.txt  conf.env*   example.odt  librarything.tsv  tmp/
$ bib import-lo
Problem with setup: SQLSTATE[HY000] [2002] Connection refused
make: *** [/opt/libre-bib/bin/bib-cmd.mak:100: status/import-lo.date] Error 1

Open another terminal:

$ ssh myserver
Enter passphrase for key '/home/bob/.ssh/id.mysys':

bob@mxlinux:/home/bob
$

Minimize the terminal window.

$ bib import-lo

Nothing was output. Edit conf.env and change cgVerbose to true.
$ bib import-lo
/opt/libre-bib/bin/import-txt-2-lo.php -c
Verbose is on.
Backup is on.
UseRemote is on.
UseLib is on.
Problem with setup: Missing: cgDbPassCache tmp/.pass.tmp. To set it,
run: bib connect [89]
make: *** [/opt/libre-bib/bin/bib-cmd.mak:100: status/import-lo.date] Error 1
$ bib connect
read -srp 'Password (fav-pet)? '; \
echo $REPLY >tmp/.pass.tmp
Password (fav-pet)? First define tunnel: ssh HOST.example.com
See: /home/bob/ssh/config
show databases; use DBNAME; show tables;

if [[ "true" == "true" ]]; then \
    tPort=3308; \
else \
    tPort=3306; \
fi; \
mysql -P $tPort -u example --password=$(cat tmp/.pass.tmp)
    -h 127.0.0.1 biblio_example
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 784
Server version: 10.5.18-MariaDB-0+deb11u1 Debian 11

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [biblio_example]> quit
Bye

(Clearly I need to cleanup the outputs.)

$ bib import-lo
/opt/libre-bib/bin/import-txt-2-lo.php -c
Verbose is on.
Backup is on.
UseRemote is on.
UseLib is on.
.
Processed 292 lines. [263]
Inserted 31 records. [264]
/opt/libre-bib/bin/convert-lo-2-bib.php -c
Verbose is on.
Backup is on.
UseRemote is on.
UseLib is on.

Processed: 31 [221]
date +%F_%T >status/import-lo.date

This imported the biblio.txt file, creating the "lo" table. You can run "bib connect" and use sql commands to look the table. For example:

show tables;
show fields from table lo;
select Identifier,Booktitle from table lo;

Now let's import the export from LibraryThing.

$ bib import-lib
librarything schema and import
/opt/libre-bib/bin/import-tsv-2-lib-db.php -c
Verbose is on.
Backup is on.
UseRemote is on.
UseLib is on.
............
Processed: 12
date +%F_%T >status/import-lib.date
head -n 1 librarything.tsv | sed 's/ /_/g' >tmp/lib-schema.tsv
diff /opt/libre-bib/etc/lib-schema.tsv tmp/lib-schema.tsv
Warning: If there are differences, there could be problems.
$ bib update-lo
Update lo from lib where Titles are similar, first 40 char
Run this after lib-db, lo-db
...................
Processed: 19
...........
Processed: 11
Created: bib_2023-05-17_01-40-14

Processed: 31 [221]

This will have created a join table with Titles are in the "lo" and "lib" tables. It then updated some empty "lo" fields from the "lib" data. For example: Publisher is tricky one. (Enhancement: Provide an option so some "lib" values will override the 'lo' values.)

$ bib backup-lo
cp: cannot stat 'backup/backup-lo.csv': No such file or directory
...............................
Processed: 31
$ ls backup/
backup-lo.csv
$ bib ref-new
Unpack example.odt [319]
Start processing [292]

Processed 1056 lines. [303]
Found 2 references. [304]
Backup example.odt [339]
Final clean-up with tidy [343]
Repack example.odt [354]
Done. [386]

This updated the REF tags so they are now biblio entries. You'll also see the original example.odt was copied to the backup/ dir.

Run: libreoffice to see how they have changed.

$ libreoffice example.odt

If you run import-lo or import-lib with updated entries, then run ref-update to update them in the example.odt file. If you add new REFs to the document then you would run ref-new again.

$ bib ref-update
Verbose is on.
Backup is on.
UseRemote is on.
UseLib is on.
Unpack example.odt [330]
Start processing [303]

Processed 1065 lines. [314]
Found 2 references. [315]
Backup example.odt [350]
Final clean-up with tidy [354]
Repack example.odt [365]
Done. [396]

Now you can add the Bibliography to the end of your document, and setup the styles for the different Type of entries.


E. Build

TBD

See the Makefile in github project.


F. Maps

The best source for the maps can be found in bin/util.php.

bib to libreoffice names

This has some minor differences when looking at the field in the Bibliography style section.

lo-file to lo-table

This maps the lo text file Tag names to the lo-table field names.

lo-table to bib-table

Do some simple formatting of the lo-table values and put them in the bib-table, so that the Bibliography style is easily setup.

lib-table to lo-table

This maps the LibraryThing field names to the Libreoffice Bibliography field names.


G. Support