Dar Documentation

Libdar APplication Interface (API) tutorial

(API version 6.x.x)

Presentation

The Libdar library provides a complete abstraction layer for handling Dar archives. The general operations provided are:

Note that Disk ARchive and libdar have been released under the Gnu General Public License (GPL). All code linked to libdar (statically or dynamically), must also be covered by the GPL or compatible license. Commercial use is prohibited unless otherwise explicitely agreeded with libdar's author.

This tutorial will show you how to use the libdar API. Since release 2.0.0 the dar command-line executable also relies on this API, looking at it's code may provide a good illustration on the way to use libdar, the file src/dar_suite/dar.cpp is the primary consumer of the libdar API. However we will see here, step by step how you can leverage the API for your own use.

In the following sample codes will be provided, they are solely illustrative and is not guaranteed to compile. Reference documentation for this API is contained in the source code as doxygen comment which can be extracted and "compiled" in the the doc/html directory when compiling dar/libdar. This API reference document is also available online and will be referred below as the API reference documentation.

Let's Start

Conventions

Language

Dar and libdar are written in C++, and so is the libdar fundamental API. Though you can find a Python binding in the src/python subdirectory, which strongly follows the class hierarchy that will see here with C++ sample code. The libdar_test.py document rawly follows with python code what we will see below with C++.

Feel free to contribute if you want bindings for other languages.

Header files

Only one include file is required in your program to have access to libdar:

#include <dar/libdar.hpp>

Libdar namespace

All libdar symbols are defined within the libdar namespace. You can either add the using namespace libdar; statement at the beginning of your source files...

using namespace libdar; get_version(); ...

...or, as shown below, you can explicitly use the namespace in front of libdar symbols, we will use this notation in the following, which has the advantage of avoiding name conflicts and clarify origin of the symbols used, but lead to a more heavier code, less easy to read:

libdar::get_version(); ...

Exceptions

The library makes use of exception to report unexpected conditions. These contain the reason and context the error occurred in, so you can be catch these in your code to display details of the error and its cause. All exceptions used within libdar inherit from the pure virtual class libdar::Egeneric

Most of the time you will use only one of the following two methods:

get_message()
returns a message string describing the error met
dump_str()
returns a text paragraph with additional information about the stack as well as context the error occurred in.

Now, messages are for human, you may need to provide different behaviors depending on the type of the error libdar has met and which triggers such exception. This is can be done by checking the type of the exception.

We will only focus on most common exception type read the API reference documentation for an exhaustive list of the exception type used by libdar:

libdar::Ebug
This one is used when a-situation-that-should-never-occur is met and can be assumed to be a bug in libdar. Using the get_message() method in that situation would not provide all necessary details to understand and fix the bug, so it is advised to always use dump_str() for that specific type of exception and abort further execution regarding libdar.
libdar::Erange
A parameter or a value is out of range, details is provided with get_message()
libdar::Ememory
Libdar lacked of virtual memory to proceed to the requested operation
libdar::Einfinint
an arythmetic error occurred when using libdar::infinint object, which is a internal integer type that can handle arbitrary large numbers. Today it is only used if libdar has been compiled with --enable-mode=infinint so you are not likely to meet it.
libdar::Elimitint
when infinint are not used, a wrapper class over system integers is used to detect integer overflow. In that situation, this exception is thrown, enjoying the user to use a infinint based flavor of libdar to avoid this error
libdar::Ehardware
used when the operating system returned an I/O error or hardware related error
libdar::Esystem
When software related error is reported by the system, like lack of ownership, permission or non existing file...
libdar::Euser_abort
This exception is thrown as the consequence of user request to abort the process. This may occur when aswnering to a question issued by libdar.
libda::Ethread_cancel
used when a program has called the thread cancellation routine of libdar, which drives libdar to stop as soon as possible (immediately when the control point is met, or delayed aborting cleanly the operation under process depending on the option used to cancel the running libdar thread.

Some other exist and are well described in the API reference documentation. You can thus wrap the whole libdar interaction with a statement like in the following example:

 try  {   // calls to libdar   ...   //  }  catch(libdar::Ebug & e)  {   std::string msg = e.dump_str();     // do something with msg like for example:   std::cerr << msg << std::endl;  }  catch(libdar::Euser_abort & e)  {   // some specific treatment for this   // type of exception   ...  }  catch(libdar::Egeneric & e)  {   // a global treatment for all other libdar exceptions   std::string msg = e.get_message(); << std::endl;   ...  }

First we must initialize libdar

the libdar initialization is performed by calling the libdar::get_version() function.

This function can be called several time though only once is necessary, but this call has to complete before any other call to libdar.

In a multi-thread context libthreadar initialization is not re-entrant. In other word the first call to libdar::get_version() must complete before any other call to libdar can take place in another thread of the running process. Once libdar has been initialized, you can call libdar::get_version() concurrently from several threads at the same time without problem.

libdar::get_version();

We should prepare the end right now

Libdar used some data-structures (mutex, secured memory, etc.) that need to be released properly before ending the process. It is important to invoke the close_and_clean() function before exiting your program if you had called get_version() previously. After that, the only allowed call to libdar is get_version().

libdar::close_and_clean()
Note:
closes_and_clean() makes the necessary for memory to be released in the proper order. Not calling close_and_clean() at the end of your program may result in uncaught exception message from libdar at the end of the execution. This depends on the compiler, libc and option activated in libdar at compilation time.

All in one, at the highest level, you code should look like the following:

 libdar::get_version();  try  {   try   {     // calls to libdar   // thing we will see in next   ...   ...   }   catch(libdar::Ebug & e)   {   std::string msg = e.dump_str();     // do something with msg like for example:   std::cerr << msg << std::endl;   }   catch(libdar::Egeneric & e)   {   std::string msg = e.get_message();     // do something with msg like for example   std::cerr << msg << std::endl;   }  }  catch(...)  {   libdar::close_and_clean();   throw;  }  libdar::close_and_clean();

Intercepting signals

libdar by itself does not make use of any signal (see signal(2) and kill(2)). However, gpgme library with which libdar may be linked with in order to support asymmetrical strong encryption (i.e. encryption using public/private keys) may trigger the PIPE signal. Your application shall thus either ignore it (signal(SIGPIPE, SIG_IGN)) or provide an adhoc handle. By default the PIPE signal leads the receiving process to terminate.

Libdar classes

The main components of libdar are four classes:

In the following we will first see class libdar::archive which will take most of our effort as other classes which we will see at the end are very trivial.

Multithreaded environment

Except when explicitely mentioned, a given libdar object can only be manipulated by a single thread. You can however perform several operations concurrently from different thread, each having its own set of libdar objects. Though, if one thread is creating an archive by mean of an first object and at the same time another thread by mean of a second object is trying to read the same archive under construction, things might not work as expected. But this is obvious considerations we will not dig any further assuming you know what you are doing.

Let's create a simple archive

Creating a libdar::archive object depending on the constructor used, leads to either:

Basic archive creation

For archive creation the constructor format is the following one:

 archive::archive(const std::shared_ptr<user_interaction> & dialog,   const path & fs_root,   const path & sauv_path,   const std::string & filename,   const std::string & extension,   const archive_options_create & options,   statistics * progressive_report);

For now we will left beside some parameters, that we will see in detail later:

Once the object has been created (the constructor has returned), the archive operation has completed and a new backup has been completely written on disk.

 libdar::archive my_first_backup(nullptr,   "/home/me",   "/tmp",   "first_backup",   "dar",   libdar::archive_options_create(),   std::nullptr);

The previous example has create a single sliced archive first_backup.1.dar located under /tmp. This backup contains the content of the directory /home/me and its sub-directories, without compression and without ciphering. You have guessed: compression, slicing, ciphering can be set by playing with passing an adhoc archive_option_create object to this archive constructor, something we will see in details a bit further.

Once the object my_first_backup has been created there is only little thing we can do with it: only archive listing or archive isolation. For archive extraction, testing or diffing, we needs creating a new object with a "read" constructor.

If we had allocated the archive on the heap (using new), we would have just added the delete invocation right after the construction of the my_first_backup object:

 libdar::archive* my_first_backup = new libdar::archive(nullptr,   "/home/me",   "/tmp",   "first_backup",   "dar",   libdar::archive_options_create(),   nullptr);     // we assume std::bad_alloc would be thrown if an allocation   // problem had occurred     // same thing if libdar throws an exception at constructor time,   // the object would not be created and would not have to be deleted.     // So now we can delete the created object:    delete my_first_backup;

Progressive report

During the operation we get nothing shown unless an error occurs. To provide more visibility on the process we will use an libdar::statistics object passed as last argument of this constructor. Then we will use some interesting method of class libdar::statistics:

If you have a doubt about the meaning and use of a particular counter in a particular operation, please refer to API reference documentation for class libdar::statistics, the private fields corresponding to these counter are explicitly defined there.

 libdar::statistics stats;    libdar::archive my_first_backup(nullptr,   "/home/me",   "/tmp",   "first_backup",   "dar",   libdar::archive_options_create(),   & stats);     // in another thread we can see the progression:    std::cout << stats.get_treated_str()   << " file(s) saved" << std::endl;    std::cout << stats.get_errored_str()   << " file(s) failed to backup" << std::endl;    std::cout << stats.get_ea_treated_str()   << " Extended Attributes saved" < std::endl;

Archive creation options

in the previous example, we have created an temporary object of class libdar::archive_options_create and passed it on-fly to the archive constructor without modifying it. Thus we used the default options for this operations. But a lot of options are available, each one can be modified by a specific method of this class Follow is a subset of the available options. We won't details them all, but you can refer the doxygen documentation for class libdar::archive_options_create for additional information.

First you may have find some new libdar types (=classes) in arguments, we will briefly explain how to set them:

std::shared_ptr<libdar::archive>

C++11 shared smart-pointer to an existing libdar::archive object. We will see how to use it next when performing differential backup

libdar::infinint

can be set from a classical unsigned int, unsigned long or other unsigned integer type, so for you this is like another unsigned integer type

libdar::mask

This class is a top of a class hierarchy containing several classes provided with libdar. It allows you to define a filtering mecanism for the feature wher it is used. We will see how to use it in a further paragraph of this tutorial.

libdar::compression

It is an enumeration which values are (among others not listed here):


libdar::U_I

A the opposite of the libdar::infinint this is not a class but an alias to the system classical unsigned integer. Depending on the operating system and CPU this might point to unsigned long or unsigned long long or other equivalent type.

libdar::crypto_algo

This is an enumeration with values like:


libdar::secu_string

This is a class used to securely storing password and sensible cryptographic information. It can be setup from a char* or better, from a filedescriptor. its main constructor is:


libdar::comparison_fields

This is an enumeration with values like:

Follows a variant of the previous backup creation example, here we set some options to a given value:

 libdar::archive_options_create opt;    opt.set_allow_over(false);   // forbids slice overwriting    opt.set_display_finished(true);   // show a summary after each completed directory    opt.set_slicing(1024000,   2000);   // slices of 1000 kiB initial slice of 2000 bytes    opt.set_pause(2);   // pause every two slices    opt.set_execute("echo slice %N completed");   // command executed after each slice    opt.set_crypto_algo(libdar::crypto_algo::aes256);     // providing an empty secu_string leads dar   // interactively ask the passphrase in a secure manner  opt.set_crypto_pass(secu_string());   // But this previous call is useless as en empty secu_string is the default   // though one could have setup a secu_string from a std::string   // this way:  std::string my_pass("hello world!");  libdar::secu_string my_secupass(my_pass.c_str(), my_pass.size());  opt.set_crypto_pass(my_secupass);    opt.set_compression(libdar::compression::xz);  opt.set_compression_level(6);  opt.set_min_compr_size(10240);  // not trying compressing file smaller than 10 kiB     // now we have the opt option object ready   // we can proceed to the archive creation using it:    libdar::archive my_first_backup(nullptr,   "/home/me",   "/tmp",   "first_backup",   "dar",   opt,   nullptr);

And of course, you can use both libdar::statistics and libdar::archive_options_create at the same time, when creating a backup

Creating a differential or incremental backup

Maybe you have guessed? Compared to the previous operation (full backup) doing an differential or incremental backup will only ask to open, in read mode, an existing archive and pass this object as argument of class archive_options_create::set_reference() seen just above.

The read-only constructor for class archive is the following:

archive::archive(const std::shared_ptr<user_interaction> & dialog, const path & chem, const std::string & basename, const std::string & extension, const archive_options_read & options);

same as before:


  // first we open the previously created archive in read mode:    std::shared_ptr<libdar::archive> ref_archive;    ref_archive = std::make_shared<libdar::archive>(nullptr,   "/home/me",   "first_backup",   "dar",   archive_create_options_read());     // for clarity we separated the creation of the   // archive object used as archive of reference from   // the archive object which creation will perform   // differential backup (see below). We could have made   // in one step using an anonymous temporary object   // (in place of ref_archive)   // invoking directly std::make_shared   // when calling set_reference() below:    libdar::archive_options_create opt;    opt.set_reference(ref_archive);    libdar::archive my_second_backup(nullptr,   "/home/me",   "/tmp",   "diff_backup",   "dar",   opt,   nullptr);

creating a incremental backup is exactly the same, the difference is the nature of the archive of reference. We used to describe a differential backup one that has a full backup as reference, while an incremental backup has another incremental or differential backup as reference (not a full backup).

Archive listing

Archive listing operation consist of the creation of an archive object in read mode as we just did above for ref_archive and invoking a method on that newly object to see all or a sub-directory content of the archive. Before looking at the listing method let's zoom on the class libdar::archive_create_options_read which we just skip over previously.

Archive reading options

The same as the class archive_options_create detailed above, the class archive_options_read has a constructor without argument that sets the different options to their default value. You can change them one by one by mean of specific methods. The most usual ones are:

set_execute() runs a command before reading a new slice of the archive. See API reference documentation for details. You will meet class archive_options_create in order to test an archive, compare an archive with filesystem, isolate an archive and repair an archive.

Listing methods

There is several ways to read an given archive contents:

  1. making use of a callback function that will be called in turn for each entry of the archive even special entries that flag the end of a directory and the next entry will be located in the parent directory:

     void op_listing(libdar::archive_listing_callback callback,   void *context,   const libdar::archive_options_listing & options) const;
  2. using the same callback but only for the different entries of a given directory, directory that has to exist in the archive of course. It returns false when the end of the directory has been reached:

     bool get_children_of(libdar::archive_listing_callback callback,   void *context,   const std::string & dir,   bool fetch_ea = false);
  3. like previous listing a given directory content but returning a vector of objects libdar::list_entry that provide detailed information about each entry, no callback is used here:

     const std::vector<libdar::list_entry> get_children_in_table(const std::string & dir,   bool fetch_ea = false) const;

For the two first methods you have to define a callback function of the following form:

 void (*)(const std::string & the_path,   const list_entry & entry,   void *context);

This callback will receive as argument the full path of the object, a libdar::list_entry object providing much details on it and the "context" value passed as argument of archive::op_listing() or archive::get_children_of()

In the following example we will use only a few methods of class libdar::list_entry that are available to get detail of a given entry of an archive, feel free to explore this class's documentation for to get all details:

  // we first create a read-mode archive object that will be used   // in the three following examples,    libdar::archive_options_read opt;    opt.set_info_details(true);  opt.set_execute("echo 'about to read slice %p/%b.%N.%e with context %c'");    libdar::archive my_backup(nullptr, // this is user_interaction we will see further   "/home/me",   "diff_backup",   "dar",   opt);
    // we will also illustrate here the use of libdar::archive_options_read   // inside the callback function we will define here:    void my_listing_callback(const std::string & the_path,   const libdar::list_entry & entry,   void *context)  {   std::cout << the_path;   if(entry.is_dir())   std::cout << " is a directory";   std::cout << " with permission " << entry.get_perm();   std::cout << " file located in slices " << entry.get_slices().display();   std::cout << std::endl;   // yep, we do not need context, this   // is available if you need it though     if(entry.is_eod())   {   // only op_listing() provides such type of object   // which occurs when we reached the End Of Directory   // next entry will be located in the parent directory.   //   // Note for op_listing: when reading a directory we recurs in it,   // meaning that the next entry this callback will be   // invoked for will be located in that directory   //   // for get_children_of() no recursion or eod object is   // performed about directory. The next entry following   // a directory is still located in the same parent directory   // which when fully read stops the get_children_of() routine   // at the difference of op_listing() which parse the whole   // directory tree.   //   // For example, reading a empty directory will provide   // that directory info, then an eod object a the next   // callback invocation.   }  }

These two objects my_backup and my_listing_callback() we just defined will be used in the following examples.

Archive listing using archive::op_listing()

First possibility: we can pass nullptr as callback function to archive::op_listing, all will be displayed in stdout

 my_backup.op_listing(nullptr, // no callback function   nullptr, // we don't use the context here   archive_options_listing()) // and use default listing options

Second possibility: we use the callback defined previously:

 my_backup.op_listing(my_listing_callback,   nullptr, // we still don't use the context here   archive_options_listing()) // and still the default listing options

In complement of both previous variant we can of course set non default listing options

 libdar::archive_options_listing opt;    opt.set_filter_unsaved(true);   // skip entry that have not been saved since the archive of reference    opt.set_slice_location(true);   // necessary if we want to have slicing information available in the callback function    opt.set_fetch_ea(false);   // this is the default. Set it to true if you   // want to use list_entry::get_ea_reset_read()/get_ea_next_read()    my_backup.op_listing(my_listing_callback,   nullptr, // we still don't care of context here   opt); // and still the default listing options

Archive listing using archive::get_children_of()

  // With this method we only list one directory    my_backup.get_children_of(my_listing_callback,   nullptr, // we still don't care of context here   "", // we read the root directory of the archive   true); // and ask for EA retrieval, but as we do not   // use list_entry::get_ea_read_next() in the   // callback this is just wasting CPU and memory       // or course if you have a sub-directory /home/me/.gnupg/private-keys-v1.d   // in your home directory and you want to check how it is saved in the   // archive, as we defined the root of the backup as /home/me and as you   // always have to pass a relative path (no leading /) you could do that by   // calling the following:    my_backup.get_children_of(my_listing_callback,   nullptr,   ".gnupg/private-keys-v1.d");

Archive listing using archive::get_children_in_table()

  // still listing a single directory but this time without callback function:    my_backup.init_catalogue();   // necessary to fill read the whole catalogue in memory   // in particular if archive has been opened in sequential read mode    std::vector<libdar::list_entry> result = my_backup.get_children_in_table(".gnupg/private-keys-v1.d");     // now reading the std::vector    std::vector<libdar::list_entry>::iterator it = result.begin();  while(it != result.end())  {   if(it->is_dir())   std::cout << " is a directory";   std::cout << " with permission " << it->get_perm();   std::cout << " located in slices " << it->get_slices().display();   std::cout << std::endl;  }

Testing an archive

As seen for listing operation we assume a archive object has been create in read mode. Testing the coherence of the relative archive files on disk is done by calling the libdar::op_test method:

 libdar::statistics op_test(const libdar::archive_options_test & options,   libdar::statistics * progressive_report);

You may have recognized the libdar::statistics type we saw for archive creation. It is present as argument and the provided libdar::statistics object can be read during the whole testing operation by another thread. But if you just want the to know the result, you'd better just use the returned value as it makes the operation quicker due to the absence of multithread management.

  // for the exercise, we will change some default options:    archive_options_test opt;  opt.set_info_details(true); // to have a verbose output    libdar::statistics stats;  stats = my_backup.op_test(nullptr, // still the user_interaction we will see further   opt; // the non default options set above   nullptr); // we will just use the returned value    std::cout << stats.get_treated_str() << "file(s) tested" << std::endl;  std::cout << stats.get_errored_str() << " file(s) with errors" << std::endl;  std::cout << stats.get_ea_treated_str() << " Extended Attributes tested" << std::endl;

Comparing an archive

As simple as previously, but using the archive::op_diff method:

 statistics op_diff(const path & fs_root,   const archive_options_diff & options,   statistics * progressive_report);

Over the type of the option field, you see the fs_root argument which define which directory of the filesystem to compare the archive to

  // for the exercise, we will change some default options:    archive_options_diff opt;  opt.set_info_details(true); // to have a verbose output    opt.set_what_to_check(libdar::comparison_fields::ignore_owner);   // this option above will consider equal two files which   // only change due to user or group ownership difference   // by default any difference will be considered a difference    (void)my_backup.op_diff("/home/me",   opt; // the non default options set above   nullptr); // not using it for this example

Isolating an archive

Still as simple as previously, but using the archive::op_isolate method:

 void op_isolate(const path &sauv_path,   const std::string & filename,   const std::string & extension,   const archive_options_isolate & options);

You will find similitude with the archive creation though here this is not a constructor

  // for the exercise, we will change some default options:    archive_options_isolate opt;  opt.set_warn_over(false);     // by default overwriting is allowed by a warning is issued first   // here overwriting will take place without warning    opt.set_compression(libdar::compression::gzip);  opt.set_compression_level(9);   // this is the default  opt.set_min_compr_size(10240);   // not trying compressing file smaller than 10 kiB    my_backup.op_isolate("/tmp",   "CAT_diff_backup",   "dar",   opt); // the non default options set above     // have you noted? There is no libdar statistics field returned nor as argument.

Restoring files from an archive

Quite as simple as previously, here we use the archive::op_extractmethod:

 statistics op_extract(const path & fs_root,   const archive_options_extract & options,   statistics *progressive_report);

  // as we still do not have seen masks, we will restore all files contained in the backup
  // such mask would be provided to the
  // archive_options_extract::set_selection() and/or
  // to the archive_options_extract::set_subtree() methods
  // to precisely define what files to restore
   archive_options_extract opt;  opt.set_dirty_behavior(false, false); // dirty files are not restored    (void) my_backup.op_extract("/home/me/my_home_copy",   opt,   nullptr); // we have seen previously how to use statistics

Merging archives

Here we will need two archive objects open in read-mode and we will invoke a specific archive constructor passing these two objects as argument, once the constructor will have completed the merging operation will be done.

 archive(const std::shared_ptr<user_interaction> & dialog,   const path & sauv_path,   std::shared_ptr<archive> ref_arch1,   const std::string & filename,   const std::string & extension,   const archive_options_merge & options,   statistics * progressive_report);

  // assuming you have two backups:   // the first is /tmp/home_backup.*.dar   // the second is /var/tmp/system_backup.*.dar   // we will create /tmp/merged.*.dar as result of the merging   // of these two backups     // 1 - first things first: opening the first backup    libdar::archive_options_read opt;    opt.set_info_details(true);  opt.set_execute("echo 'about to read slice %p/%b.%N.%e with context %c'");  std::shared_ptr<libdar::archive> home_backup(new libdar::archive(nullptr, // this is user_interaction we'll see further   "/tmp",   "home_backup",   "dar",   opt));     // 2 - opening the second backup    std::shared_ptr<libdar::archive> system_backup(new libdar::archive(nullptr,   "/var/tmp",   "system_backup",   "dar",   opt);     // 3 - setting up the options for merging operation    libdar::archive_options_merge opt_merge;    opt_merge.set_auxiliary_ref(system_backup);   // while merging the second backup is optional, where from the use of option for it  opt_merge.set_slicing(1048576, 0); // all slice would have 1 MiB at most  opt_merge.set_compression(libdar::compression::bzip2);  opt_merge.set_keep_compressed(true);  opt_merge.set_user_comment("archive resulting of the merging of home_backup and system_backup");  opt_merge.set_hash_algo(libdar::hash_algo::sha512); // will generate on-fly hash file for each slice     // 4 - now performing the merging operation
   libdar::archive merged(nullptr, // still the user_interaction we will see further   "/tmp",   home_backup, // first back is mandatory, not part of options   "merged",   "dar",   opt_merge,   nullptr); // progressive_report, we don't use here

Decremental backup

Decremental backup is an operation that, from two full backups, an old and a recent one, creates a backward differential backup corresponding to the old full backup, which difference is based on the new full backup. In other words, instead of keeping two full backups, you can keep the latest and replace the oldest by its decremental counterpart. This will save you space while letting you restore as if you had the old full backup by restoring first the recent (full) backup then the decremental backup.

Creating a decremental backup is exactly the same as creating a merging backup, you need just to set the archive_options_merge::set_decremental_mode() before proceeding to the merging. To avoid duplication we will just illustrate the last step of the previous operation modified for decremental backup:

  // creating the two read-only backup as for merging operation   // the only difference is that here both backups are mandatory  std::shared_ptr<libdar::archive> old_full_backup(...); // not detailing this part  std::shared_ptr<libdar::archive> new_full_backup(...); // not detailing this part     // setting up the options for a decremental operation  libdar::archive_options_merge opt_merge;    opt_merge.set_decremental_mode(true);  opt_merge.set_auxiliary_ref(new_full_backup);     // now performing the merging operation (here decremental backup)    libdar::archive merged(nullptr, // still the user_interaction we will see further   "/tmp",   old_full_backup,   "decremental_backup",   "dar",   opt_merge,   nullptr); // progressive_report we don't use here

Archive repairing

If an archive has been truncated due to lack of disk space and if sequential marks (aka tape marks) had not been disable, it is possible to rebuild sane archive beside this truncated one.

We just need to invoke a specific libdar::archive constructor which form follows:

 archive(const std::shared_ptr<user_interaction> & dialog,   const path & chem_src,   const std::string & basename_src,   const std::string & extension_src,   const archive_options_read & options_read,   const path & chem_dst,   const std::string & basename_dst,   const std::string & extension_dst,   const archive_options_repair & options_repair);

You should now be familiarized with the different types and variable uses. As you can note, this constructor takes in charge the work to read the damaged archive because if the archive is corrupted a normal constructor would fail, so you won't have to do it first. As always, this constructor will end only once the operation will have completed, that's to say at the end of the reparing.

  // assuming the archive /tmp/home_backup.*.dar is damaged   // and you want to have repaired archive as /tmp/home_backup_repaired.*.dar    libdar::archive repaired(nullptr, // still the user_interaction we have not yet seen   "/tmp"   "home_backup",   "dar",   archive_options_read(),   "/tmp",   "home_backup_repaired",   "dar",   archive_options_repair());     // we have not done fancy things with the two option classes, but we did above   // enough time for you get all the necessary information from the API reference   // documentation

Looking at some details

we have covered the different operations the class libdar::archive can be used for, still remains some concepts to view:

Then we will see the three other more simple classes:

This will be the subject of the following chapters, but for now, maybe you remember that we had to initialize libdar before use, by calling libdar::get_version()? This routine also exists with arguments that will provide as its name suggests the libdar version:

void get_version(U_I & major, U_I & medium, U_I & minor, bool init_libgcrypt = true);

It is advised to use this form to fetch the libdar version major, medium and minor numbers to be sure the library you've dynamically linked with is compatible with the features you will be using:

If you use libgcrypt beside libdar in your application you should initialize libgcrypt and not let it be done by libdar the latest argument of this form should be set to false in that case, according to the libgcrypt documentation which indicates that libgcrypt should normally (always) be initialized directly from the application not from an intermediate library.

Follows an example of test that can be performed while initializing libdar:

 U_I major, medium, minor;    libdar::get_version(major, medium, minor);    if(maj != libdar::LIBDAR_COMPILE_TIME_MAJOR ||   med < libdar::LIBDAR_COMPILE_TIME_MEDIUM)  {   std::cout << "libdar version we link with is too old for this code" << std::endl;   // throw an exception or anything else appropriate to that condition:   throw "something";  }

Checking compile-time features activation

Once we have called one of the get_version* function it is possible to access the list of features activated at compilation time thanks to a set of function located in the compile_time nested namespace inside libdar:

 void my_sample_function()  {   bool ea = libdar::compile_time::ea();   bool largefile = libdar::compile_time::largefile();   bool nodump = libdar::compile_time::nodump();   bool special_alloc = libdar::compile_time::special_alloc();   U_I bits = libdar::compile_time::bits();   // bits is equal to zero for infinint,   // else it is equal to 32 or 64 depending on   // the compilation mode used.     bool thread = libdar::compile_time::thread_safe();   bool libz = libdar::compile_time::libz();   bool libbz2 = libdar::compile_time::libbz2();   bool liblzo = libdar::compile_time::liblzo();   bool libxz = libdar::compile_time::libxz();   bool libcrypto = libdar::compile_time::libgcrypt();   bool furtive_read = libdar::compile_time::furtive_read();   // for details see the compile_time namespace in the API reference documentation  }

User Interaction

we have seen std::shared_pointer on class libdar::user_interaction previously but did not used this feature.

Defining your own user_interaction class

class libdar::user_interaction defines the way libdar interact with the user during an operation, like an archive creation, restoration, testing and so on. Only four types of interaction are used by libdar:

void message(const std::string & message); void pause(const std::string & message); std::string get_string(const std::string & message, bool echo); secu_string get_secu_string(const std::string & message, bool echo);

By default an inherited class of libdar::user_interaction, called libdar::shell_interaction, is used and implements these four type of exchange by mean of text terminal:

For a GUI you will probably not want stdin and stdout to be used. Instead of that you have the possibility to implement your own inherited class from user_interaction. This one should look like the following:

 class my_user_interaction: public libdar::user_interaction  {   protected:   // display of informational message   virtual void inherited_message(const std::string & message) override;     // display of a question and returns the answer from user as true/false   virtual bool inherited_pause(const std::string & message) override;     // display the message and returns a string from the user,   // with or without display what the user typed (echo)   virtual std::string inherited_get_string(const std::string & message, bool echo) override;     // same as the previous be the user provided string is returned as secu_string   virtual libdar::secu_string inherited_get_secu_string(const std::string & message, bool echo) override;  };

Relying on the pre-defined user_interaction_callback class

As an alternative to defining your own inherited class from libdar::user_interaction, libdar provides a class called user_interaction_callback which is an implementation of the user interaction, based on callback functions.

You will need to implement four callback functions:

using message_callback = void (*)(const std::string &x, void *context); using pause_callback = bool (*)(const std::string &x, void *context); using get_string_callback = std::string (*)(const std::string &x, bool echo, void *context); using get_secu_string_callback = secu_string (*)(const std::string & x, bool echo, void *context);

Then you can create an libdar::user_interaction_callback object using this constructor:

 user_interaction_callback(message_callback x_message_callback,   pause_callback x_answer_callback,   get_string_callback x_string_callback,   get_secu_string_callback x_secu_string_callback,   void *context_value);

Here follows an example of use:

 void my_message_cb(const std::string & x, void *context)  {   std::cout << x << std::endl;  }    bool void my_pause_cb(const std::string & x, void *context)  {   char a;     std::cout << x << endl;   std::cin >> a;   return a == 'y';  }    std::string my_string_cb(const std::string & x, bool echo, void *context)  {   // to be defined  }    libdar::secu_string my_secu_string_cb(const std::string & x, bool echo, void *context)  {   // to be defined  }     // eventually using a context_value that will be passed to the callback of the object  void *context_value = (void *)(& some_datastructure);    std::shared_ptr<libdar::user_interaction> my_user_interaction(new libdar::user_interaction_callback(my_message_cb,   my_pause_cb,   my_string_cb,   my_secu_string_cb,   context_value));

You can also find the predefined classes libdar::user_interaction_blind which always says no in name of the user displays nothing and provide empty strings, as well as libdar::shell_interaction_emulator which given a user_interaction object send to it the formatted information as if it was a shell_interaction object, leading one to emulate libdar default behavior under any type of "terminal".

IMPORTANT
all libdar::user_interaction inherited classes provided by libdar are not designed to be manipulated by more than one thread at a time. The use of std::shared_ptr is only here to let the caller not have to manage such object and let libdar release it when no more needed or to let the caller reuse the same user_interaction object for a subsequent call to libdar which would not be possible if a std::unique_ptr was used instead.

Now if you design your own user_interaction inherited class and provide them mecanism (mutex, ...) that allow them to be used simultaneously by several thread there is no issue to pass a such object as argument to different libdar object used by different threads running at the same time.

Masks

Mask are used to define which string will be considered and which will not. Libdar implements masks as several classes that all inherit from a virtual class named libdar::mask that defines the way masks are used (interface class). This class defines the bool mask::is_covered(const std::string & expression) const method which libdar uses to determine whether a given string matches or not a mask and thus whether the corresponding entry (filename, EA, directory path depending on the context), is eligible or not for an operation.

Strings applied to masks may correspond to filename only, to full path or maybe to other things (like Extended Attributes). That's in the context where the mask is used that the string meaning take place, thing we will see further.

There is several different basic masks classes you can use to build fairly complex masks, while it is possible you should not need to define you own mask classes, if the need arises, please contact libdar developer if you thing an additional class should take place beside the following ones:

class name behavior
class libdar::bool_mask boolean mask, either always true or false (depending on the boolean passed at its constructor), it matches either all or none of the submitted strings
class libdar::simple_mask matches strings as done by the shell on the command lines (see "man 7 glob")
class libdar::regular_mask matches regular expressions (see "man 7 regex")
class libdar::not_mask negation of another mask (the mask given at construction time)
class libdar::et_mask makes an *AND* operator between two or more masks
class libdar::ou_mask makes the *OR* operator between two or more masks
class lbdar::simple_path_mask matches whether the string to evaluate is subdirectory of, or is the directory itself that has been given at construction time.
class libdar::same_path_mask matches if the string is exactly the given mask (no wild card expression)
class libdar::exclude_dir_mask matches if string is the given string or a sub directory of it
class libdar::mask_list matches a list of files defined in a given file

Let's play with some masks:

  // all files will be elected by this mask  libdar::bool_mask m1(true);     // all string that match the glob expression "A*~" will match.   // the second argument of the constructor tell whether the match is case sensitive so here,   // any file beginning by 'A' or by 'a' and ending by '~' will be selected by this mask:  libdar::simple_mask m2(std::string("A*~"), false);     // m3 is the negation if m2. This mask will thus match   // any string that does not begin by 'A' or 'a' or does not finishing by '~'  libdar::not_mask m3(m2);     // this mask matches any string that is a subdirectory of "/home/joe"   // and any directory that contains /home/joe, meaning   // "/", "/home", "/jome/joe" and any subdirectory are matched.   // here, the second argument is also case sensitivity (so   // "/HoMe" will not be selected by this mask as we set it to "true".  libdar::simple_path_mask m4 = simple_path_mask("/home/joe",   true);     // now let's do some more complex things:   // m5 will now match only strings that are selected by both m2 AND m4  libdar::et_mask m5;  m5.add_mask(m2);  m5.add_mask(m4);     // we can make more interesting things like this, where m5 will select files   // that match m2 AND m4 AND m3. But as m3 is not(m2), m5 will never   // match any file... but the idea here is to see the flexibility of use:  m5.add_mask(m3);     // but we could do the same with an "ou_mask" and would get a silly   // counterpart of m1 (a mask that matches any files)  libdar::ou_mask m6;  m6.add_mask(m2);  m6.add_mask(m4);  m6.add_mask(m3);     // lastly, the NOT, AND and OR operation can be used recursively.   // Frankly, it's even possible to have masks referring each other!  libdar::not_mask m7(m6);  m6.add_mask(m7);   // now that's to you to build something that makes sense...

The idea here is not to create object manually, but to link their creation to the action and choices the user makes from the user interface (Graphical User Interface of your application, for example)

Now that you've seen the power of these masks, you should know that in libdar masks are used at several places:

An exception is the archive testing operation, which has no fs_root argument (because the operation is not relative to an existing filesystem), however the subtree argument exist to receive a mask for comparing the path of file to include or exclude from the testing operation. In this case the situation is as if the fs_root was set to the value "<ROOT>". For example, masks will be compared to "<ROOT>/some/file" when performing an archive test operation.

Instead of using explicit string "<ROOT>" you can use libdar::PSEUDO_ROOT predifined std::string variable

Aborting an Operation

If the POSIX thread support is available, libdar will be built in a thread-safe manner, giving you the possibility to have several threads using libdar at the same time (but on different objects except concerning the libdar::statistics which can be shared between threads). You may then wish to interrupt a given thread. But aborting a thread form the outside (like sending it a KILL signal) will most of the time let some memory allocated or even worse can lead to dead-lock situation, when the killed thread was inside a critical section and had not got the opportunity to release a mutex. For that reason, libdar proposes a set of calls to abort any processing libdar call which is ran by a given thread.

  // next is the thread ID in which we want to have lidbar call canceled   // here for simplicity we don't describe the way the ID has been obtained   // but it could be for example the result of a call to pthread_self() as   // defined in <pthread.h> system header file  pthread_t thread_id = 161720;     // the most simple call is:  libdar::cancel_thread(thread_id);     // this will make any libdar call in this thread be canceled immediately     // but you can use something a bit more interesting:  libdar::cancel_thread(thread_id, false);     // this second argument is true for immediate cancellation and can be ommited in   // that case. But using false instead leads to a delayed cancellation,   // in which case libdar aborts the operation   // but produces something usable, espetially if you were performing a backup.     // You then get a real usable archive which only contains files saved so far, in place   // of having a broken archive which misses a catalogue at the end. Note that this   // delayed cancellation needs a bit more time to complete, depending on the   // size of the archive under process.

As seen above, cancellation can be very simple. What now succeeds when you ask for a cancellation? Well, an exception of type Ethread_cancel is thrown. All along his path, memory is released and mutex are freed. Last, the exception appears to the libdar caller. So, you can catch it to define a specific comportment. And if you don't want to use exceptions a special returned code is used.

 try  {   libdar::archive my_arch(...);   ...  }  catch(libdar::Ethread_cancel & e)  {   ... do something specific when thread has been canceled;
 }

Some helper routines are available to know the cancellation status for a particular thread or to abort a cancellation process if it has not yet been engaged.

 thread_t tid;     // how to know if the thread tid is under cancellation process?  if(libdar::cancel_status(tid))   std::cout << "thread cancellation is under progress for thread : "   << tid << std::endl;  else   std::cout << "no thread cancellation is under progress for thread : "   << std::endl;     // how to cancel a pending thread cancellation ?  if(libdar::cancel_clear(tid))   std::cout << "pending thread cancellation has been reset, thread "   << tid << " has not been canceled"   << std::endl;  else   std::cout << "too late, could not avoid thread cancellation for thread "   << tid   << std::endl;

Last point, back to the libdar::Ethread_cancel exception, this class has two methods you may find useful, when you catch it:

 try  {   ... some libdar calls  }  catch(libdar::Ethread_cancel & e)  {   if(e.immediate_cancel())   std::cout << "cancel_thread() has been called with \"true\" as second argument"   << std::endl;   else   std::cout << "cancel_thread() has been called with \"false\" as second argument"   << std::endl;     U64 flag = e.get_flag();   ... do something with the flag variable...  }     // what is this flag stored in this exception?
  // You must consider that the complete definition of cancel_thread() is the following:   // void cancel_thread(pthread_t tid, bool immediate = true, U_64 flag = 0);   // thus, any argument given in third is passed to the thrown Ethread_cancel exception,   // value which can be retrieved thanks to its get_flag() method. The value given to this   // flag is not used by libdar itself, it is a facility for user program to have the possibility   // to include additional information about the thread cancellation.     // supposing the thread cancellation has been invoked by:  libdar::cancel_thread(thread_id, true, 19);   // then the flag variable in the catch() statement above would have received   // the value 19.

Dar_manager API

For more about dar_manager, please read the man page where are described in detail its available features.

To get dar_manager features you need to use the class database. Most of the methods of the database class make use options. The same as what has been seen with class archive a auxiliary class is used to carry these options.

Database object construction

Two constructor are available. The first creates a brand-new but empty database in memory:

database(const std::shared_ptr<user_interaction> & dialog);

As seen for libdar::archive dialog can be set to a null pointer if the default interaction mode (stdin/stdout/stderr) suits your need.

The second constructor opens an existing database from filesystem and stores its contents into memory ready for further use and actions:

 database(const std::shared_ptr<user_interaction> & dialog,   const std::string & base,   const database_open_options & opt);

Here follows simple examples of use of class database:

 std::shared_ptr<libdar::user_interaction> ui_ptr; // points to null  libdar::database base(ui_ptr);   // we have created an empty database (no archive in it) called "base"    libdar::database other(ui_ptr, // we can reuse it as it points to nullptr   "/tmp/existing_base.dmd",   libdar::database_open_options());   // we have created a database object called "other" which contains   // (in RAM) all information that were contained in the   // database file "/tmp/existing_base.dmd"    libdar::database_open_option opt;  opt.set_partial(true);  opt.set_warn_order(false);  libdar::database other2(ui_ptr,   "/tmp/existing_base.dmd",   opt);     // we have created yet another database object called "other2" which differs   // from "other" by the option we used. While "other" is a fully loaded   // database, "other2" is a partial database. This notion is explained   // below

In the following we will indicate whether a database operation can be applied to a partially loaded database or not. All operation can be applied to a fully loaded databse.

Database's methods

A database can be open in read-write mode partially loaded (still read-write) mode and last in partially loaded read-only mode. All operations are available in the first mode, but some are not in the second and even less in the third mode. We will detail which one are available in each mode:

Available in partially loaded read-only mode

Availabler in partially loaded read-write mode

Available in fully loaded read-write mode

Well, you might now say that as description this is a bit light for a tutorial, yes. In fact these call are really very simple to use, you can find a complete description in the API reference documentation. This documentation is built if doxygen is available and is put under doc/html after calling make in the source package. It is also available from dar's homepage.

Dar_slave API

dar_slave role is to read an archive while interacting with a dar process through a pair of pipes. Dar asks portion of the archive or information about the archive in the first pipe from dar to dar_slave. And dar_slave sends the requested information into the other pipe toward dar (embedded into an expected format).

Since API 6.0.x, dar_slave has an API. It is implemented by the class libdar::libdar_slave. You need firs to create an object using the following constructor:

libdar_slave(std::shared_ptr<user_interaction> & dialog, const std::string & folder, const std::string & basename, const std::string & extension, bool input_pipe_is_fd, const std::string & input_pipe, bool output_pipe_is_fd, const std::string & output_pipe, const std::string & execute, const infinint & min_digits);

Once the object is created, you will need to call the libdar_slave::run() method which will end when the dar process at the other end will no more need of this slave:

 libdar::libdar_slave slave(nullptr,   "/tmp",   "first_backup",   "dar",   false,   "/tmp/toslave", // assuming this is an existing named pipe   false,   "/tmp/todar", // assuming this is also an existing named pipe   "echo 'reading slice %p/%b.%N.%e in context %c'",   0);    slave.run();     // once run() has returned, you can launch it again for another process   // it will continue to provide access to the /tmp/first_backup.*.dar archive

Dar_xform API

dar_xform creates a copy of a given archive modifying its slicing. it does not require decompressing nor deciphering the archive to do so. There is different constructor depending whether the archive is read from filesystem, from a named pipe of from a provided file descriptor

Reading from a file

 libdar::libdar_xform(const std::shared_ptr<user_interaction> & ui,   const std::string & chem,   const std::string & basename,   const std::string & extension,   const infinint & min_digits,   const std::string & execute);

Reading from a named pipe

 libdar_xform(const std::shared_ptr<user_interaction> & dialog,   const std::string & pipename);

Reading from a file descriptor

 libdar_xform(const std::shared_ptr<user_interaction> & dialog,   int filedescriptor);

Creating a single or multi-sliced archive on filesystem

Once the libdar::libdar_xform object is created it can copy the referred archive to another location in another form thanks to one of the two libdar_xform::xform_to methods. There is not link between the constructor used and the libdar_xform::xform_to flavor used, any combination is possible.

 void xform_to(const std::string & path,   const std::string & basename,   const std::string & extension,   bool allow_over,   bool warn_over,   const infinint & pause,   const infinint & first_slice_size,   const infinint & slice_size,   const std::string & slice_perm,   const std::string & slice_user,   const std::string & slice_group,   libdar::hash_algo hash,   const libdar::infinint & min_digits,   const std::string & execute);

Creating a single sliced archive toward a filedescriptor

 void xform_to(int filedescriptor,   const std::string & execute);

Here follows an example of use. We will convert a possibly multi-sliced archive to a single slice one, generating a sha512 hash file on-fly.

 std::shared_ptr<libdar::user_interaction> ui_ptr; // points to null  libdar::libdar_xform transform(ui_ptr,   "/tmp",   "my_first_archive",   "dar",   0,   "echo 'reading slice %p/%b.%N.%e context is %c'");    transform.xform_to("/tmp",   "my_other_first_archive",   "dar",   false, // no overwriting allowed   true, // does not matter whether we warn or not as we do not allow overwriting   0, // no pause between slices   0, // no specific first slice   0, // no slicing at all (previous argument is thus not used anyway in that case)   "", // using default permission for created slices   "", // using default user ownership for created slices   "", // using default group ownership for created slices   libdar::hash_algo::sha512, // the hash algo to use (for no hashing use hash_none instead)   0, // min_digits ... not using this feature here where from we use "0"   "echo 'Slice %p/%b.%N.%e has been written. Context is %c'");

Compilation & Linking

Compilation

All the symbols found in the libdar API defined from <dar/libdar.h>. So you should only need to include this header. If the header file is not located in a standard directory, in order to compile your code, you may need some extra flags to pass to the compiler (like -I/opt/...). The pkg-config tool can help here to avoid system dependent invocation:

 shell prompt > cat my_prog.cpp    #include <dar/libdar.h>    main()  {   libdar::get_version(...);   ...  }    shell prompt > gcc `pkg-config --cflags libdar` -c my_prog.cpp

Linking

Of course, you need to link your program with libdar. This is done by adding -ldar plus other library libdar can rely on like libz, libbzip2, liblzo or libgcrypt, depending on the feature activated at compilation time. Here too, pkg-config can provide a great help to avoid having system dependent invocation:

shell prompt > gcc pkg-confg --libs libdar` my_prog.o -o my_prog

Libdar's different flavors

The compilation and linking steps described above assume you have a "full" libdar library. but beside the full (alias infinint) libdar flavor, libdar also comes in 32 and 64 bits versions. In these last ones, in place of internally relying on a special type (which is a C++ class called infinint) to handle arbitrary large integers, libdar32 relies on 32 bits integers and libdar64 relies on 64 bits integers (there are limitations which are described in doc/LIMITATIONS). But all these libdar version (infinint, 32bits, 64bits) have the same interface and must be used the same way, except for compilation and linking:

These different libdar versions can coexist on the same system, they share the same include files. But the LIBDAR_MODE macro must be set to 32 or 64 when compiling or linking with libdar32 or libdar64 respectively, this macro changes the way the libdar headers files are interpreted by the compiler. pkg-config --cflags will set the correct LIBDAR_MODE, so you should only bother calling it with either libdar, libdar32 or libdar64 depending on your need: "pkg-confg --cflags libdar64" for example.

shell prompt > cat my_prog.cpp #include <dar/libdar.h> main() {   libdar::get_version(...);   ... } shell prompt > gcc -c `pkg-config --cflags libdar64` my_prog.cpp shell prompt > gcc `pkg-config --libs libdar64` my_prog.o -o my_prog>

and replace 64 by 32 to link with libdar32.