lanties
AVForums Super Veteran
Mods - This is a lot of info. You are welcome to remove if you so wish.
Forum members - Only read this if you are looking for a tool to organise your FLAC files.
Introducing Lanties FLAC Organiser - (LFO)
=============================================================================
Lfo is a standalone application with a Graphical User Interface (GUI), developed by myself, using
Lazarus and Free Pascal. There is a Windows as well as a Linux version available. They are identical.
The main function is to assist me to organise my FLAC library into a "lanties best practice directory
hierarchy". At some stage I will convert it for a headless environment where the GUI will be replaced
with a web interface allowing me to run it directly on my LMS Linux server. The GUI was just easier
and quicker to test. All libraries are separate from the GUI so it should be relatively straight forward.
No 3rd party FLAC libraries have been used. All development was done from scratch.
Background
=============================================================================
Having ripped many CDs over many years to FLAC using different ripping software with different
path topologies has left me with my music being all over the place. In many cases I also ended up
with many duplicates as the children would rip and copy their FLAC files to our music server. With
more than 10,000 CDs and 130,000 tracks on my server it has become time to sort my music out. I
have always been able to walk up to my CD rack and almost instantly locate the right CD. I would
very much like to be able to do the same again with an on-disk-structure (ODS) that works for me.
Purpose
=============================================================================
1. Move my FLAC files to locations on my music server that make sense to me where I can
easily find the albums I am looking for.
2. Get rid of duplicates irrespective of destination directories where the actual files will be
located.
3. Fix poorly tagged FLAC files and cover art.
4. Backup my music across servers in order to synchronise more than one music server on a
regular bases.
Why developed lfo when there is a trillion other similar programs?
=============================================================================
Because I am a hacker and therefore I can.
Your right to use and my disclaimer
=============================================================================
You are welcome to use lfo if it will benefit you in any way. I will however not take any responsibility
for any losses as a result of your decision to use lfo. I strongly recommend that you make backups
before using any functionality of this program. You also should have the original CDs so the risk is
not that great now or is it? I have not written a GUI program in more than 10 years so please forgive
the lack of modern innovative ideas in design and implementation.
PM me if you would like a copy and I will send a dropbox link where you can download it from.
What lfo is not?
=============================================================================
It is most certainly not a catchall for everything FLAC file related. It is not a bulk metadata tagger but
I do like my Gracenote (www.gracenote.com) integration and the tag editor. Lfo is not very
configurable. You cannot select your own tags to create structures for output paths. You have to
accept my logic or convince me to change mine. See explanation of logic further down. It is not a CD
ripper. It is not an open source project. It is most probably not "bug free".
What is required to run lfo?
=============================================================================
Windows 7 or Linux Desktop like Lubuntu on i386.
OpenSSL to access Gracenote. I do provide the latest dll's (hopefully without heartbleed issues) with
the windows version. You will need to install openssl vir your Linux system. It is currently only 32bits.
I have not bothered to compile it under 64bit bit I have a good feeling it will compile first time. (I am
using it on Windows 64 at the moment)
Multi-threaded
=============================================================================
Lfo is a multi-threaded application and the only thing you can actually configure at the moment is
the maximum number of threads that can run at any given point in time. The default is 8 threads but
you will have to experiment with what works for you on your hardware. There is always a point of
diminishing returns when disk IO becomes too congested and the threads may actually become very
slow. Copying 8 files at a time, even on a I7, can suck your resources dry.
Installing lfo
=============================================================================
Just copy the files to a directory of your choice and run the executable. I am really not interested to
write an installation program. Lfo works very well if you have at least two monitors like all decent
programmers should.
The actual process of using lfo to restructure files and get rid of duplicates
=============================================================================
1. Select input path - This is where your current flac files are. I suggest use a small sample set
first just for testing.
2. Select output path - This is where the files are going to be copied to. The relative paths will
be created below this output path.
3. Generate relative paths. (See Duplicate logic below) This process runs through your flac files
and use the tags to make up strings that represent the relative paths. It will also generate a
list of duplicates found from this run. Note that no directories are created or files copied in
this process. This can take a bit of time depending on the number of files you have as it
reads the Metadata from every single file to construct the relative path.
4. If you have any duplicates or funnies in your relative paths fix the tags or remove the
duplicates using the tools in lfo or any other file manager or tag editor of your choice but
remember the tags that lfo uses.
5. If you had to do 4 above clear your relative path cache and repeat from 3 until you have no
duplicates or funnies in your relative paths.
6. Copy files
7. Done! No check the results and do not remove the originals unless you are 100% sure all is
OK!!
Duplicate track management logic
=============================================================================
Ok this may still fail me but this is the current logic that I use. Hopefully if this do fail, I will be able to
use lfo to fix the failure and get rid of the duplicates. The whole concept is based on what I call
"relative path indexing" created from the tags in the flac file. This immediately indicates that the
tags that I use in my path must be correct. Hopefully lfo can also make it easy to find these files with
poor tags etc. A relative path will take the following format:-
First Letter of Album Artist/Album Artist/Sample rate/Album Name/Track Number - Track Title +
'.flac'
A relative path is constructed from the following tags:
1. Compilation
2. Album Artist
3. Album Name
4. Track Number
5. Track Title
6. Sample Rate - Not a tag. I get it from StreamInfo
7. Ensemble
Compilation - If this tag is present and the value is 1 I will use VA instead of the Album Artist to VA
(Various Artists). No further checking is done on Album Artist name.
Album Artist - If this tag is present I will use this for the Album Artist otherwise I will use the Artist
tag. If the Album Artist contains the words "Various Artists" or "Verskeie" I will use VA instead of the
Album Artist. If the tag ensemble is present I will use the value from this tag as the Album Artist.
Case is not important in any of the tags. I also remove or convert any "funny" characters in the
relative path to make sure that all relative paths can ultimately be converted into physical disk paths.
Paths are correct for Windows vs Linux OS \ vs /. This is what a real relative path would look like
under Windows:
N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
And a album from a compilation
\V\VA\44100\Cafe D`afrque Lll\ 01 - Ice Flowers.flac
So here is the absolute essence of this basic logic. FLAC files with the same tags, going through the
same lfo "filters" will result in the same relative paths and be considered duplicates. This obviously
exclude the same FLAC files that are in compilations. For example if Nirvana's Smells Like Teen Spirit
was the number one track in a compilation of for example Memorable Rock Songs, it would look like
this:
\V\VA\44100\ Memorable Rock Songs \01 - Smells Like Teen Spirit.flac
Thus not considered a duplicate by lfo. It is important to understand that lfo can and should detect
duplicates even if there are different base paths. For example
C:\dad\ N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
Vs
C:\children\N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
Is considered a duplicate file by lfo.
Updating Tag logic
=============================================================================
Currently I only have two concepts (replace and merge) but I can already think of more. Replace is
self-explanatory. Merge is basically as follows: - Use lfo's tags and add the existing tags from the
track that is not already defined by lfo tags. I can see that I will soon have to add some more options
here. (Scratching my head)
Update Cover logic
=============================================================================
Simple - Replace or not. You can also load a cover images from disk. I do not touch pictures
embedded in flac files. Personally I believe this is bad practice. Why compress to flac and then add
the some cover to every track when all music servers understand and work with cover.jpg or
folder.jpg.
Copy vs Move
=============================================================================
I currently only copy files from source to destinations. I know move can be faster on the same drives
but I would rather be safe than sorry. Please note that when lfo updates tags it is actually recreating
a new flac file and deleting the old one. I do try and preserve the file date and time. Losses can occur
if something goes wrong at this stage. But as all good people I know you should still have your
original CDs
I will also never replace a file that is already in the destination folder. The advantage
of this is that should you stop copying you can continue at a later stage. I would however
recommend that you wait till all copy threads are finish before you close lfo.
Character conversions in creating file paths form tag values
=============================================================================
1. All accent characters are converted to the non-accent version e.g. '?' is converted to 'e'.
2. '&' is replaced with 'and'.
3. All illegal filename characters are converted to '_' e.g. '?' is converted to '_'.
Search filter logic
=============================================================================
The filter mechanism is constantly filtering down in the current result set. To start a new filter press
Reset to load the original result set in. Filters will only work on relative paths. So you must generate
relative paths before trying filters. All changes done to relative paths in a filter process will be lost
when you click reset. If the changes were as a result of TAG changes just run "Generate Relative
Paths" again.
Gracenote integration and TAGS in general
=============================================================================
First things first. To use this function you have to register for Gracenote. This is a Gracenote
requirement, not lfo. To do this select Tags -> Register on Gracenote. If it is faded out it means you
are already registered. I know I can do this automatically but I prefer not to make internet "calls"
without the user's permission. The registration is autonomous but I do believe Gracenote keeps
track of how many request lfo is making as there is an application key as well as a user key that is
combined in all requests for information to Gracenote.
You can add and remove tags. Duplicate tags are allowed. There are minimum tags that cannot be
removed or altered. There are two tag editors. The one on the main form and the one on the
Gracenote Tag form. Both work slightly different but both has only one purpose and that is to fix
tags. The Autocomplete function on the Tag form will fill all empty tags with the most used, even if it
is just one, tags in the album. E.g. if you want the set the Genre to Jazz you only need to do it for one
track and then click AutoComplete and it will set the same genre for all other tracks. Currently I only
retrieve one level of tags from Gracenote. Not sure If I want and or need more. I may also add
support for other Meta database engines in future but Gracenote is for free for free applications like
lfo and it seems to have support for many South African artists. I have noticed that I sometimes have
to request data more than once. So if your initial search does not deliver tags try again or change
track and try again. I search Gracenote with Album, Artist and Track Title. I found the "Single-Best"
option from Gracenote not working well with some of my Albums and hence I return a number of
Albums. You will need to select the right one. Please make sure it is the right one. Lfo adds two tags
lfo=modified date and GNID=grace note track ID if tags come from Gracenote. Not required but I
thought this may become handy at some state.
In sharing lfo logic
=============================================================================
I hope lfo can help you as much as it is currently helping me to manage and sort out my flac
collection. Sharing an application like this and your thoughts around the design exposes certain flaws
in one's own way of thinking and doing things. I understand and accept this. I am however not
interested in explaining why I am doing certain things in specific ways more than what I have done
here already. You are welcome to make suggestions with motivations and I will consider building
them in, but please do not ask me to explain or defend my way of thinking. After all my first priority
with lfo is to serve my own needs. If it can also help you then great!
Kind regards
Lanties
Forum members - Only read this if you are looking for a tool to organise your FLAC files.
Introducing Lanties FLAC Organiser - (LFO)
=============================================================================
Lfo is a standalone application with a Graphical User Interface (GUI), developed by myself, using
Lazarus and Free Pascal. There is a Windows as well as a Linux version available. They are identical.
The main function is to assist me to organise my FLAC library into a "lanties best practice directory
hierarchy". At some stage I will convert it for a headless environment where the GUI will be replaced
with a web interface allowing me to run it directly on my LMS Linux server. The GUI was just easier
and quicker to test. All libraries are separate from the GUI so it should be relatively straight forward.
No 3rd party FLAC libraries have been used. All development was done from scratch.
Background
=============================================================================
Having ripped many CDs over many years to FLAC using different ripping software with different
path topologies has left me with my music being all over the place. In many cases I also ended up
with many duplicates as the children would rip and copy their FLAC files to our music server. With
more than 10,000 CDs and 130,000 tracks on my server it has become time to sort my music out. I
have always been able to walk up to my CD rack and almost instantly locate the right CD. I would
very much like to be able to do the same again with an on-disk-structure (ODS) that works for me.
Purpose
=============================================================================
1. Move my FLAC files to locations on my music server that make sense to me where I can
easily find the albums I am looking for.
2. Get rid of duplicates irrespective of destination directories where the actual files will be
located.
3. Fix poorly tagged FLAC files and cover art.
4. Backup my music across servers in order to synchronise more than one music server on a
regular bases.
Why developed lfo when there is a trillion other similar programs?
=============================================================================
Because I am a hacker and therefore I can.
Your right to use and my disclaimer
=============================================================================
You are welcome to use lfo if it will benefit you in any way. I will however not take any responsibility
for any losses as a result of your decision to use lfo. I strongly recommend that you make backups
before using any functionality of this program. You also should have the original CDs so the risk is
not that great now or is it? I have not written a GUI program in more than 10 years so please forgive
the lack of modern innovative ideas in design and implementation.
PM me if you would like a copy and I will send a dropbox link where you can download it from.
What lfo is not?
=============================================================================
It is most certainly not a catchall for everything FLAC file related. It is not a bulk metadata tagger but
I do like my Gracenote (www.gracenote.com) integration and the tag editor. Lfo is not very
configurable. You cannot select your own tags to create structures for output paths. You have to
accept my logic or convince me to change mine. See explanation of logic further down. It is not a CD
ripper. It is not an open source project. It is most probably not "bug free".
What is required to run lfo?
=============================================================================
Windows 7 or Linux Desktop like Lubuntu on i386.
OpenSSL to access Gracenote. I do provide the latest dll's (hopefully without heartbleed issues) with
the windows version. You will need to install openssl vir your Linux system. It is currently only 32bits.
I have not bothered to compile it under 64bit bit I have a good feeling it will compile first time. (I am
using it on Windows 64 at the moment)
Multi-threaded
=============================================================================
Lfo is a multi-threaded application and the only thing you can actually configure at the moment is
the maximum number of threads that can run at any given point in time. The default is 8 threads but
you will have to experiment with what works for you on your hardware. There is always a point of
diminishing returns when disk IO becomes too congested and the threads may actually become very
slow. Copying 8 files at a time, even on a I7, can suck your resources dry.
Installing lfo
=============================================================================
Just copy the files to a directory of your choice and run the executable. I am really not interested to
write an installation program. Lfo works very well if you have at least two monitors like all decent
programmers should.
The actual process of using lfo to restructure files and get rid of duplicates
=============================================================================
1. Select input path - This is where your current flac files are. I suggest use a small sample set
first just for testing.
2. Select output path - This is where the files are going to be copied to. The relative paths will
be created below this output path.
3. Generate relative paths. (See Duplicate logic below) This process runs through your flac files
and use the tags to make up strings that represent the relative paths. It will also generate a
list of duplicates found from this run. Note that no directories are created or files copied in
this process. This can take a bit of time depending on the number of files you have as it
reads the Metadata from every single file to construct the relative path.
4. If you have any duplicates or funnies in your relative paths fix the tags or remove the
duplicates using the tools in lfo or any other file manager or tag editor of your choice but
remember the tags that lfo uses.
5. If you had to do 4 above clear your relative path cache and repeat from 3 until you have no
duplicates or funnies in your relative paths.
6. Copy files
7. Done! No check the results and do not remove the originals unless you are 100% sure all is
OK!!
Duplicate track management logic
=============================================================================
Ok this may still fail me but this is the current logic that I use. Hopefully if this do fail, I will be able to
use lfo to fix the failure and get rid of the duplicates. The whole concept is based on what I call
"relative path indexing" created from the tags in the flac file. This immediately indicates that the
tags that I use in my path must be correct. Hopefully lfo can also make it easy to find these files with
poor tags etc. A relative path will take the following format:-
First Letter of Album Artist/Album Artist/Sample rate/Album Name/Track Number - Track Title +
'.flac'
A relative path is constructed from the following tags:
1. Compilation
2. Album Artist
3. Album Name
4. Track Number
5. Track Title
6. Sample Rate - Not a tag. I get it from StreamInfo
7. Ensemble
Compilation - If this tag is present and the value is 1 I will use VA instead of the Album Artist to VA
(Various Artists). No further checking is done on Album Artist name.
Album Artist - If this tag is present I will use this for the Album Artist otherwise I will use the Artist
tag. If the Album Artist contains the words "Various Artists" or "Verskeie" I will use VA instead of the
Album Artist. If the tag ensemble is present I will use the value from this tag as the Album Artist.
Case is not important in any of the tags. I also remove or convert any "funny" characters in the
relative path to make sure that all relative paths can ultimately be converted into physical disk paths.
Paths are correct for Windows vs Linux OS \ vs /. This is what a real relative path would look like
under Windows:
N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
And a album from a compilation
\V\VA\44100\Cafe D`afrque Lll\ 01 - Ice Flowers.flac
So here is the absolute essence of this basic logic. FLAC files with the same tags, going through the
same lfo "filters" will result in the same relative paths and be considered duplicates. This obviously
exclude the same FLAC files that are in compilations. For example if Nirvana's Smells Like Teen Spirit
was the number one track in a compilation of for example Memorable Rock Songs, it would look like
this:
\V\VA\44100\ Memorable Rock Songs \01 - Smells Like Teen Spirit.flac
Thus not considered a duplicate by lfo. It is important to understand that lfo can and should detect
duplicates even if there are different base paths. For example
C:\dad\ N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
Vs
C:\children\N\Nirvana\44100\Nevermind, Disc 1\ 01 - Smells Like Teen Spirit.flac
Is considered a duplicate file by lfo.
Updating Tag logic
=============================================================================
Currently I only have two concepts (replace and merge) but I can already think of more. Replace is
self-explanatory. Merge is basically as follows: - Use lfo's tags and add the existing tags from the
track that is not already defined by lfo tags. I can see that I will soon have to add some more options
here. (Scratching my head)
Update Cover logic
=============================================================================
Simple - Replace or not. You can also load a cover images from disk. I do not touch pictures
embedded in flac files. Personally I believe this is bad practice. Why compress to flac and then add
the some cover to every track when all music servers understand and work with cover.jpg or
folder.jpg.
Copy vs Move
=============================================================================
I currently only copy files from source to destinations. I know move can be faster on the same drives
but I would rather be safe than sorry. Please note that when lfo updates tags it is actually recreating
a new flac file and deleting the old one. I do try and preserve the file date and time. Losses can occur
if something goes wrong at this stage. But as all good people I know you should still have your
original CDs
I will also never replace a file that is already in the destination folder. The advantage of this is that should you stop copying you can continue at a later stage. I would however
recommend that you wait till all copy threads are finish before you close lfo.
Character conversions in creating file paths form tag values
=============================================================================
1. All accent characters are converted to the non-accent version e.g. '?' is converted to 'e'.
2. '&' is replaced with 'and'.
3. All illegal filename characters are converted to '_' e.g. '?' is converted to '_'.
Search filter logic
=============================================================================
The filter mechanism is constantly filtering down in the current result set. To start a new filter press
Reset to load the original result set in. Filters will only work on relative paths. So you must generate
relative paths before trying filters. All changes done to relative paths in a filter process will be lost
when you click reset. If the changes were as a result of TAG changes just run "Generate Relative
Paths" again.
Gracenote integration and TAGS in general
=============================================================================
First things first. To use this function you have to register for Gracenote. This is a Gracenote
requirement, not lfo. To do this select Tags -> Register on Gracenote. If it is faded out it means you
are already registered. I know I can do this automatically but I prefer not to make internet "calls"
without the user's permission. The registration is autonomous but I do believe Gracenote keeps
track of how many request lfo is making as there is an application key as well as a user key that is
combined in all requests for information to Gracenote.
You can add and remove tags. Duplicate tags are allowed. There are minimum tags that cannot be
removed or altered. There are two tag editors. The one on the main form and the one on the
Gracenote Tag form. Both work slightly different but both has only one purpose and that is to fix
tags. The Autocomplete function on the Tag form will fill all empty tags with the most used, even if it
is just one, tags in the album. E.g. if you want the set the Genre to Jazz you only need to do it for one
track and then click AutoComplete and it will set the same genre for all other tracks. Currently I only
retrieve one level of tags from Gracenote. Not sure If I want and or need more. I may also add
support for other Meta database engines in future but Gracenote is for free for free applications like
lfo and it seems to have support for many South African artists. I have noticed that I sometimes have
to request data more than once. So if your initial search does not deliver tags try again or change
track and try again. I search Gracenote with Album, Artist and Track Title. I found the "Single-Best"
option from Gracenote not working well with some of my Albums and hence I return a number of
Albums. You will need to select the right one. Please make sure it is the right one. Lfo adds two tags
lfo=modified date and GNID=grace note track ID if tags come from Gracenote. Not required but I
thought this may become handy at some state.
In sharing lfo logic
=============================================================================
I hope lfo can help you as much as it is currently helping me to manage and sort out my flac
collection. Sharing an application like this and your thoughts around the design exposes certain flaws
in one's own way of thinking and doing things. I understand and accept this. I am however not
interested in explaining why I am doing certain things in specific ways more than what I have done
here already. You are welcome to make suggestions with motivations and I will consider building
them in, but please do not ask me to explain or defend my way of thinking. After all my first priority
with lfo is to serve my own needs. If it can also help you then great!
Kind regards
Lanties
