Release notes for Retroshare 0.6.7

This release comes with multiple changes and bug fixes, resulting of 2 years of development. A lot of effort has been made to get this release really stable.

Web Interface

We ship Retroshare v0.6.7 with a new Web interface, which has been developed over 2-3 rounds of Google Summer of Code. It is mostly useful for controling a Retroshare node running on a remote/headless server. In order to start the web interface,

  1. in preferences => Web interface, check the “Enable Retroshare WEB interface”, enter a passphrase, then click “Apply settings”. Note the port here (9092 by default).
  2. open your web browser and go to http://localhost:9092/index.html (note: the “index.html” is mandatory)
  3. enter your web interface passphrase.

Installing Retroshare+Web interface to control a headless node:

  1. configure a new node on your server (either copy an existing data directory, or start retroshare Qt interface in remote and create a node), then quit;
  2. launch retroshare-service (See option -h for alternative options). Option “-W” asks for the web interface to run, “-U list” is optional and will ask you to select which node to start if you have multiple nodes available:
    ./retroshare-service/src/retroshare-service -W -U list
  3. enter a chosen password for the web interface (this is asked at every start), then choose the node you want to run (if using “-U list”), enter your profile passphrase, and wait for it to start;
  4. the web interface is only published locally. So in order to securely access the interface from a client machine, you need to create a ssh tunnel to your server, so that the web interface port will be forwarded to your server. The following line (that will not terminate) just forwards localhost:9092 of the headless server, to localhost:9092 on your client machine:
    ssh [IP of the running Retroshare node] -L 9092:localhost:9092 -N
  5. now you can access the web interface at http://localhost:9092/index.html on the client machine.

Under the hood the interface is implemented in Javascript, and uses libretroshare Json API to communicate with a running instance of the software using a secured token system.

Note: it is possible to leave retroshare-service in the background, as soon as it displays the “Retroshare core started” message. On Linux, using “Ctrl+z” and “bg” will leave the program running in the background. Do not launch it in the background right away, since the passphrase input will probably not work.

Note: the web interface is a new feature, which lacks some components of the Qt interface. It offers however the essential tools to control a headless node. It also runs with automatically configured Tor nodes.

Qt UI improvements

Lots of minor bugs have been fixed in the graphical user interface. The Qt interface is however quite dense with lots of options, although we try as much as possible to remove duplicate functionalities. Significant improvements have been made to the interface in Mail, Channels, advanced file search,… The stability of the software has also been improved quite a lot.

Running with Tor

Tor support has been moved to libretroshare. This allows retroshare-service to use Tor as well and therefore it is now possible to run headless hidden nodes. Although seamless in the Qt interface, this change needed quite a lot of development effort, since the original Tor communication system was based entirely on he Qt signal/slot system, which we decided to not depend on in libretroshare.

Retroshare moved to Tor v3 addresses, since the old v2 addresses are now deprecated. Consequently, if you have a Tor-v2 node, you’ll need to change your Tor node address to v3. This is easily done the following way:

  1. delete (or rename) the hidden service directory, [data directory]/HID06_[your ID]/hidden_service
  2. restart Retroshare. If will fail to find your missing Tor key, and will automatically generate a new one.
  3. send your new address to friends. Although you will keep your friends, your friends will not be able to contact you since your Tor address changed. This step is only mandatory when friends are clear nodes.

Also, an existing retroshare node will likely not start if using a version of Tor that is incompatible with the v3 address, and will keep displaying “Checking for hidden services:New service acquired. Status is 2: not ready yet.” in the console. In this case, upgrade Tor, or supply a v3-compatible version of Tor to the commandline using -t.

As usual, Retroshare+Tor is an incredibly powerful combination: it allows you to safely add unknown friends, and traverses NATs and firewalls much more efficiently than Retroshare alone, thanks to the network transport layer being entirely left to Tor, without a significant drop in speed.

Embedded friend server (Tor nodes only)

Retroshare+Tor now comes with an embedded friend server. The friend server is a standalone executable that accepts Tor connections on a hidden address to exchange friend certificates. It is designed to run without parameters, from the command line. In can be started as follows (on Linux. Not tested yet on windows):

/usr/bin/retroshare-friendserver

Additional arguments may include “-c [some directory]” so as to setup the place where the friend server will store its internal data (Default is “FSData/” in the current directory). Use also “-h” to get more help about parameters. After letting it start (the console will show quite a few output lines that may be used for reporting bugs), it will display the following information

======== Retroshare Friend Server as properly started =======
=                                                           =
=     Address:Port tfv2z5hquq...ytpviaod7pjeid.onion:9878   =
=                                                           =
=============================================================

That address and port should be communicated to friends who may want to use the friend server. In their own node, the information needs to be filled in the Network/Friend server tab, as shown below:

Once the address+port are entered, Retroshare will check them. If the friend server is online, it will be found and the led gets green (assuming Tor finds a route to it, which may take a few seconds). Enter your Retroshare passphrase (this is needed for internal security checks) and then check on/off. The friend server will supply friends to you automatically, and display them in the table below.

Note: The friend server is a totally experimental feature that may not be perfectly debugged yet. So please send us feedback if anything fails.

Submodule architecture

Since libretroshare is now used in multiple projects, we had to switch to a submodule architecture. This makes a lot of interesting standalone parts of the software available for use in other softwares as well. This is the case especially for libretroshare, libbitdht, and openpgp-sdk. For developers, it’s preferable to clone from a clean directory, using the following commands:

> git clone git@github.com:RetroShare/RetroShare.git
> git submodule update --init --remote --force libbitdht/ libretroshare/ openpgpsdk/

The existence of submodules implies some additional constraints when updating (e.g. submodules needs to be updated) but brings more flexibility to the developpers.

We’re currently heading toward a full cmake-based compilation. Although compiling with qmake is still possible, compiling with cmake is a new possibility that should support submodules more elegantly.

Internal improvements

A push system has been introduced into our data distribution system GXS, which allows a node to notify connected nodes when new data is available. As a result synchronization of forum posts, channel posts and votes, etc, occurs almost instantly, whereas it used to take at most 2 minutes in previous versions.

The internal storage for Mail has been improved in ways that allows the interface to be more consistent. The interface can now display the full list of destinations for each email and allows a proper “reply all” function.

The graphical user interface has been switched to using a blocking API with asynchronous calls everywhere, in replacement for the old token system. This makes the code more compact, and more elegant, and the API easier to use. The old token system is still used internally in libretroshare in order to avoid internal locking events.

The internal image storage format has been switched from PNG to JPEG, which severely reduces the data size of channel and board posts. Existing posts are not automatically converted, but new posts now use JPEG.

SSL security level

In most operating systems, OpenSSL recently switched to a default security level of 2, which forbids certificates with a SHA1 signature. Although Retroshare tries to start with a lower security level, this is not always accepted by OpenSSL. The side effect is that retrohare 0.6.7 may fail to start and ask you to create a new node (with the same profile). In this case, you’d lose friends and identities.

Identities cannot be exported, but friends can. A new identity will need to be created and signed by the same profile in the new node. In order to start the old SHA1-based node, it is possible to trick OpenSSL by creating a configuration file as containing:

# put the text below into new file as .retroshare/openssl_temp.cnf
# then run the app as normal user from console using: 

openssl_conf = default_conf

[ default_conf ]
ssl_conf = ssl_sect

[ ssl_sect ]
# temporarily enable this
system_default = system_default_sect_old
# after migrating you could check by enabling new default section:
# system_default = system_default_sect

[ system_default_sect_old ]
# SSL_CONF_CMD for abouts of CipherString
# man openssl-ciphers
CipherString = DEFAULT:@SECLEVEL=0

[ system_default_sect ]
# SSL_CONF_CMD for abouts of CipherString
# man openssl-ciphers
#MinProtocol = TLSv1.2
#CipherString = DEFAULT:@SECLEVEL=1
# or use the current default level:
CipherString = DEFAULT:@SECLEVEL=2

Then start Retroshare using

 $ OPENSSL_CONF=".retroshare/openssl_temp.cnf" retroshare 

This should allow you to export your friend list (Right-click in Network tab => Export friend list), and import it again in the new node.

Future Work

One major step forward will be to fully get rid of HTML for encoding messages. In addition to taking an unacceptable amount of space, HTML brings unnecessary security risks. Qt however uses HTML natively for displaying decorated text, possibly mixing images, text, links, etc. Consequently we’re aiming for a markup language which would only be converted to HTML for display purpose.

Another important future step is to refactor the internal chat system so as to allow private chat group discussions. The plan is currently to abstract the algorithm used for chat rooms (which works pretty reliably) and use it both in chat rooms and in distant chat using virtual peers provided by GXS tunnels. This way a separate component will take care of keeping distant tunneled connections while the routing method will distribute data among all participants.

Finally we want to get rid of openpgp-sdk, which is an old and unmaintained library. A number of possible replacements are available. The switch will be facilitated by the fact that the code in libretroshare uses an abstract class to interact with the PGP library. This may also offer opportunities for additional cryptographic primitives such as ECC (Elliptic Curve Cryptography) which are faster than RSA at comparable security levels.

Known bugs

On some linux systems (e.g. Debian 11), Retroshare used to crash when selecting the gtk2 style in preferences => Appearence. The crash can be prevented by defining the environment variable QT_QPA_PLATFORMTHEME=gtk2. Consequently, we removed the gtk2 entry from the styles list, unless this environment variable is defined when Retroshare is started. In other words, if you want gtk2 style, define the variable above in your environment.

Final word

As usual, we would like to stress that Retroshare is quite a powerful tool. It should be used wisely and in the full respect of the laws in your country.

Many thanks to Google Summer of Code and Freifunk for their incredible support, to the students and mentors who participated in the program, and to all testers and devs who contributed to this release.

The Retroshare Project now has a presence on the newly formed Libera.Chat IRC network

The Retroshare Project now has a presence on the newly formed Libera.Chat IRC network created by former FOSS Freenode staffers. Drop in and say hello, invite your friends, share you Retroshare keys and shared interests!

Libera.Chat Servers

Webchat is available at web.libera.chat/
Information on connection via Tor https://libera.chat/guides/connect#accessing-liberachat-via-tor
Matrix Gateway retroshare:libera.chat
irc.libera.chat:6697 <– Encrypted SSL Connection
irc.libera.chat:6667 <– Plaintext Connection
130.239,18.120:6697

Alternative libera.chat network servers are

tungsten.libera.chat:6697
sodium.libera.chat:6697
platnium.libera.chat:6697
palladium.libera.chat:6697
molybdenum.libera.chat
strontium.libera.chat
zirconium.libera.chat


(Use port 6667 for unencrypted, or the preferred port 6697 securely encrypted connection). Once you connect to Libera.Chat with your IRC client then you can issue

   /join #retroshare

To Register your nickname on the new Libera.Chat network you then issue

   /msg NickServ REGISTER YourPassword youremail@example.com

Libera.Chat Retroshare Chatrooms

The #retroshare chatroom is the Main public facing IRC chatroom. This is the main meet and greet chatroom for RetroShare. This IRC chatroom has a bi-directional relay to #retroshare on OFTC and to the Matrix.org retroshare room.

The #retroshare-community chatroom is a newly added IRC chatroom for the Retroshare Community at large. A community lounge for Retroshare users and their friends to exchange RS keys, share their hobbies and find shared interests. This is a more relaxed flavored community chatroom. There are no relays linked to this chat room.

Note: ##retroshare chatroom is a newly added added Offtopic and Offproject channel for visitors to play games, music, discuss politics, global warming, environmental issues etc. Statements, Opinions expressed by visitors although expected to be in a respectful manner are not necessarily reflective of the Retroshare Project itself. There are no relays linked to this chat room

The #retroshare-project chatroom is a newly added private RetroShare IRC chatroom for Retroshare founders, developers, beta-testers, builders and high level guests that are invited to share their interests with Retroshare or discuss the inner-workings of the RetroShare Project.

This private IRC chatroom has flags requiring users to have a encrypted connection, either know the key (password) to enter or be invited to enter. Under consideration would be a additional flag requiring users of this private chat room need to be registered on Libera.Chat network to enter. There are of course no relays linked to this private chat room.

Historical context

Following a internal conflict by Freenode staffers and the new Freenode owner, disgruntled staffers left Freenode to create the new FOSS Libera.Chat global network of IRC servers bringing along with a great many former Freenode parked open source big name projects and FOSS projects began migrating to Libera.Chat. Early in the initial phase of this falling out, my relations with several FOSS project developers and founders led me to begin submitting papers and requests to move the FOSS Free and Open Source Retroshare Project IRC presence from Freenode over to Libera.Chat. The Retroshare Project was awarded a Project Status by the Libera.Chat staffers along with the perks that status provides and we’ve been actively setting the #retroshare project channels up for guests and visitors.

Freenode to Libera.Chat staffers falling out sources

Freenode IRC staff quit after a new owner of Freenode LTD (allegedly) “seized” control of (the Freenode IRC) network https://boingboing.net/2021/05/19/freenode-irc-staff-quit-after-new-owner-seizes-control.html

Major open source projects abandon Freenode following an alleged “hostile” takeover https://www.techradar.com/in/news/major-open-source-projects-abandon-freenode-following-hostile-takeover

Additional #RetroShare IRC channels, rooms on other IRC Networks
OFTC, Freenode, Rizon and SpotChat

OFTC #retroshare

  • Home https://www.oftc.net/
  • Webchat https://webchat.oftc.net/
  • IRC OFTC Servers irc.oftc.net #retroshare currently attached with a 2-way relay to libera.chat #retroshare IRC chatroom.
  • ircs://irc.oftc.net:6697 for SSL (alternative port: 9999), IPv4 and IPv6.
  • irc://irc.oftc.net:6667 for non-SSL (alternative ports: 6668-6670, 7000), IPv4 and IPv6.

Freenode #retroshare

Rizon #retroshare

Spotchat #retroshare

IRC Network #retroshare search https://netsplit.de/channels/?chat=%23retroshare

Thanks again to Cyril, Dr. Bob and the RetroShare Developer core and staff, Drop in and be sure to register your nickname on the new Libera.Chat IRC Network. I look forward to visiting with you.


ASmith.

Release notes for v0.6.6

This release took even more time than the previous ones. This is for one simple reason: we changed lots of core features of the software (e.g. token system, notifications, etc), which had us re-implement some GUI parts taking the opportunity to make them more efficient or re-designed to a better layout (e.g. channels, boards). In the end, lots of bugs have been introduced in the process and it took some time to reach a perfectly stable product again. While we decided to stop developing new features in November 2020, it took us 4 months to fix all the bugs, mostly thanks to a really good user-feedback. As you will see, it was totally worth it.

1 – New channel and board layouts

Channels benefit from a new layout that is more suitable to their usage. Two different visualization modes are possible (top right blue button, left to the search field): grid or list. Use the double image below to compare both views:

Both in channels or boards, the new visualization system is now based on an abstract item model (instead of manually inserted items as it was in v0.6.5) making it much faster to load and offering a lot more possibilities for how to display the information.

2 – Short certificates

Up to v0.6.5, Retroshare used certificates containing the SSL id, the PGP public key and naming information. It appears that we can afford to only send the PGP fingerprint instead of the full key, without losing any security. Here’s why (“<=” means validated by) :

Full certificate (old method, a.k.a. type-1):

SSL handshake <= certificate PGP signature <= profile being listed as friend.

Short certificate (new method, a.k.a. type-2):

SSL handshake <= SSL Id listed as friend

In both cases the handshake is eventually validated by information provided by your friend using a supposedly secure transmission (hand-to-hand, encrypted email, etc). This part of the security model is still up to the user.

Once a node is connected, it will supply its own PGP key using discovery (as before), which will be checked to match the fingerprint in the certificate and then be marked to be used in type-1 handshake later on for future connections. The new handshake type is indeed only used at the first connection. Whatever happens wrong in the process (mismatch profile keys fingerprint, mismatched SSL id) will prevent further connections.

The security model of Retroshare connections is based on the fact that the original certificate you get comes from the right person. It can be the SSL id (new type) or the profile public key (old type): the security level is the same. If you trust the SSL id, then any information received after connecting to this Id can be trusted as well, which includes the profile public key. Once the profile key is received, all connections (including to other nodes of the same profile) will be validated by the same profile key.

The new home page therefore displays a much shorter certificate, which happens to be small enough to e.g. fit into a QR-code, since it only contains the node and profile names, the SSL Id and profile fingerprint, and the contact local/external IPs and ports (for clear nodes):

With Tor nodes, the certificate may be a bit longer since the Tor v3 onion address is rather longer (56 bytes) than the PGP fingerprint (20 bytes).

3 – Compatibility with Tor v3

Retroshare v0.6.6 supports Tor v3 keys. The change is backward compatible: new Tor v3 nodes will connect to 0.6.5 nodes without any change, provided that the 0.6.5 node you’re connected to uses a version of Tor that supports v3 keys too, which means posterior to v0.3.2.9.

Tor nodes have never been easier to setup: at the node creation time, select “Hidden node over Tor” instead of “Standard node”, and fill your profile name and password. Retroshare will automatically create a hidden service for you and configure itself to use it (as soon as Tor is installed in the system – retroshare only looks for a Tor executable). Using Tor has many advantages:

  • Connections through firewalls are extremely reliable and fast since Tor is very hard to filter. A Tor node may therefore be a good way to keep Retroshare working in a heavily filtered network;
  • your IP is hidden to everyone including friend nodes;
  • a lot of features related to clear nodes are not needed (IP Filters, Relay, DHT, UDP, etc) which makes the software lighter and certainly more stable.

Tor v2 keys will automatically deprecate after July 2021. This is not our decision: it depends on Tor stopping to support these keys. Consequently, it is recommended to upgrade your existing Tor v2 node to Tor v3. Fortunately, this is rather easy: just delete (or move away) the directory ~/.retroshare/HID06_[your SSL Id]/hidden_service/. Then start retroshare. It will automatically create a new (v3) service address. Your friends will still accept you when you connect to them (your profile key hasn’t changed). They will however not be able to locate you themselves (your Tor address has changed), until they receive your new Tor address either by discovery, or by yourself sending your certificate. This mostly happens to clear nodes for which the only direction of connection is from the clear node to the Tor node. In summary:

  • delete the hidden_service directory and re-launch Retroshare
  • send again your Retroshare certificate (See Home Page) to friend clear nodes

More information: Tor v2 deprecation timeline

4 – Auto-cleaning of unsubscribed channels, forums, boards and circles

All GXS groups (understand e.g. a forum, a channel or even a circle or an identity) now record the last time they are advertised by friends. It is therefore possible, when a group is not subscribed, to safely delete it. The rules are the following:

  • unsubscribed forums, channels, boards are auto-deleted after 60 days if not seen at friend nodes either (meaning friends are not subscribed)
  • subscribing to a group resets the counter

Some services however have specific rules:

  • Identities record the last usage time that depends on how they are used by other services. The People tab comes with a list of usages, which timings correspond to regular sweeps of the databases for signature and consistency checks.
  • Circles have specific auto-subscribe rules to consistently benefit from this auto-cleaning strategy:
    1. reminder: circle membership for an identity is based on two necessary conditions: (1) the identity is “invited” by the circle administrator (which is part of the group data), and (2) the identity requests membership (requests are group messages);
    2. a circle is only subscribed (meaning propagated to friends) if a membership request from one of your identities is posted to it, which includes also when actively “leaving” a circle after previously being a member, or if you are the administrator of that circle.
    3. circle membership requests (or leaving requests) are kept for 1 year. A membership request therefore needs to be renewed after one year.

In the early days of circles (previous releases) it was possible to create huge circles inviting all possible identities, which would automatically propagate everywhere. Thanks to the above rules, this is not possible anymore. Such circles that may already be here will be unsubscribed and disappear from your circle list after 60 days provided that none of your friend nodes request membership (The remaining time is visible in the tooltip over unsubscribed circles as “last seen: XX days ago”).

5 – Better notifications in Circles/Channels/Forums/Boards

The previously named “Log” tab is now called “Activity”. It still shows a configurable summary of events in the software (new forum messages, connection attempts, etc). We added a few of them especially for Circles:

  • Peer requesting membership or leaving a circle;
  • peer owned by your node being invited or rejected from a circle by the circle admin.

…and Forums: moderators added/removed

These notifications are now available because we added a more accurate description of the changes in group and message data in the GXS internal event handler.

6 – Various GUI tweaks

The list of small changes/improvements in the Qt interface is rather large. You will certainly notice the following incomplete subset:

  • A new tab for identities has been added in Statistics;
  • home page has been improved for more readability and a better compatibility with short certificates;
  • forum pinned posts behavior and look has been improved;
  • messages layout have been improved with spam box filter, attachments box filter and colored tags folders

7 – Certificates signed with SHA256 instead of SHA1

This is a basic security issue: primary collisions with SHA1 are not very far away and switching to a stronger hash makes SSL handshake more secure. Consequently all certificates created by Retroshare now use SHA256, while keeping compatibility with SHA1 signatures.

For the same reason, on debian Buster, openssl security level defaults to to 2 which precludes the use of SHA1, originally used to sign RS certificates. As a consequence, on these systems, locations created with RS previously to Jan. 2020 will not start unless one reverts the security level to a value of 1, which we only do when loading the certificate with default parameters fails and for OpenSSL versions later than v1.1.1. This backward compatibility tweak is meant to disappear from Retroshare code in 1 year.

SHA256 will be used automatically on new nodes (even with the same profile). Upgrading an existing node is unfortunately not currently possible.

8 – Deprecation of the token system

The existing asynchronous token system has been replaced by a blocking API, that proves much easier to use since it removes the need to have multiple functions calling and receiving events. Still, the blocking calls can take some time and the UI still needs to do them in a thread with the following constraints:

  • UI modifications should still happen in the UI thread;
  • the UI shouldn’t wait for blocking API calls.

Consequently we use a combination of two new built-in methods (that will simplify even more with C++ async calls): RSThreads::async() which executes a lambda in a separate thread and qtthreadsutils::postToObject() which sets a given lambda to be executed in the GUI thread using the normal Qt event queue. Eventually the code looks like this, as extracted from the People tab:

 void IdDetailsDialog::showIdentity(const RsGxsId& mid) {

 RsThread::async([this]()
 {
     std::set<RsGxsId> ids( { RsGxsId(mId) } ) ;
     std::vector<RsGxsIdGroup> ids_data;
 
     // call the blocking API
     if(!rsIdentity->getIdentitiesInfo(ids,ids_data))
     {
         std::cerr << "failed retrieve identity"<< std::endl;
         return;
     }
     RsGxsIdGroup group(ids_data[0]);
 
     RsQThreadUtils::postToObject( [group,this]()
     {
       /* Here it goes any code you want to be executed on the Qt Gui
        * thread, for example to update the data model with new
        * information after a blocking call to RetroShare API 
        * complete */
 
        displayIdentity(group);
     }, this );
 });
 }

Note that no mutex is needed since everything happens in the UI thread.

9 – New notification system

Adding notifications to a service is now extremely simple: a new entity named rsEvents handles all events. In order to use it, one needs to declare his own type of events, with any private information (hash and event code in the example below), and supply the serial_process method using Retroshare core macros. This way the event can be sent anywhere in libretroshare, plugins, and even to clients connected using the JSON API (e.g. web interface, python scripts, etc):

struct RsFileTransferEvent: RsEvent
{
     RsFileTransferEvent()  
       : RsEvent(RsEventType::FILE_TRANSFER),  
         mFileTransferEventCode(RsFTEventCode::UNKNOWN) {}
     ~RsFileTransferEvent() override = default; 

     void serial_process( RsGenericSerializer::SerializeJob j,
                       RsGenericSerializer::SerializeContext& ctx ) 
                          override 
     {     
         RsEvent::serial_process(j, ctx);
          RS_SERIAL_PROCESS(mFileTransferEventCode);     
         RS_SERIAL_PROCESS(mHash); 
     } 
     RsFileTransferEventCode mFileTransferEventCode; 
     RsFileHash              mHash;
 };

Whenever an event needs to be sent, the code only needs to allocate one event and send it to postEvent (asynchronous through a queue) or sendEvent (direct blocking call):

auto ev = std::make_shared<RsFileTransferEvent>();
ev->mHash = hash;
ev->mFileTransferEventCode = RsFTEventCode::DOWNLOAD_COMPLETE;
rsEvents->postEvent(ev);

Catching the event, for instance in retroshare-gui is pretty simple as well. The client (here the file transfers GUI part) only needs to register an event handler for the given event type (here FILE_TRANSFER):

mEventHandlerId=0;
rsEvents->registerEventsHandler( [this](std::shared_ptr<const RsEvent> event) 
{ 
    RsQThreadUtils::postToObject( [this,event]() 
    { 
        handleEvent_main_thread(event); 
    }) ;
}, mEventHandlerId, RsEventType::FILE_TRANSFER );

Note the use of postToObject (here again!) which allows the event to be handled in the GUI thread, since rsEvent will likely call the lambda that is registered as event handler from a different thread (here, the file transfer loop).

10 – End of retroshare-nogui (don’t panic!)

The former headless retroshare server has been replaced by a new executable named retroshare-service, to which one talks using the retroshare built-in JSON API. It is rather cheap and can be ran on Android phones as well as screenless servers.

Compilation simply requires the appropriate flags:

> qmake CONFIG="rs_jsonapi"

An example of application to this interface is the new WebUI developed during Google Summer of Code 2019. See https://github.com/retroshare/RSNewWebUI, and add “rs_webui” in the compilation flags.

While retroshare_service can be used to create new nodes, the web UI doesn’t currently offer to do it. Setting up a retroshare server with retroshare_service+WebUI therefore needs to copy the full ~/.retroshare/ data directory prior to launching retroshare service:

> ./retroshare_service/src/retroshare-service -U list -W

While ‘-U’ would normally be followed by the node ID you want to start, using ‘list’ instead will show up the list of available nodes and ask for the profile password before starting it. Another limitation is currently that retroshare_service does not use the automatic Tor configuration code (which depends on Qt) and setting up a headless Tor node requires some manual tweaking in order to (1) configure Tor system-wide to start a hidden service, then (2) create a Tor node with the Qt UI, choosing the manual Tor configuration option and setting the correct ports according to what was supplied to Tor. Option ‘-W’ asks retroshare_service to set a new password for the webinterface.

The webinterface is now entirely supplied by Retroshare. In particular its html and JS files do not need anymore to be installed on the client where the WebUI runs.

11 – Other backstage improvements

Some backstage improvements are also worth mentioning:

  • A new cache for metadata allows a much faster loading of forums/channels/boards;
  • GxsId tracking has been improved, which can be seen by more accurate usage statistics in People->Person;
  • RTT/bandwidth management and service priorities has been improved after lots of careful tests. The RTT now between two friends keeps low values;
  • memory usage has improved, but is still not perfect. Some systems tend to allocate too much memory on threads and RS uses lots of them;
  • improved debug output thanks to new RsDbg(), RsError() and RsWarn() macros;
  • consistently switched licence to AGPLv3 (GUI) LGPLv3 (libretroshare) in prevision to application for Debian packaging.

Last words

As usual, only congratulating all the testers and pull-requests providers will not make them true justice. This release would not be here without them. So many thanks to all of them!

Since we reached some rather stable status, we would now like to

  • Apply for debian packaging (we will need some sponsoring!)
  • cleanup deep indexing (currently using xapian) so that it can be shipped in the next stable version
  • improve the reactivity of GXS sync
  • make our Tor automatic system Qt-free. This is not so easy because the current code heavily depends on signal/slot connections.

Release notes for v0.6.5

Dear users, we’re proud to announce the release of RetroShare 0.6.5. This new version is coming rather late (it was originally planned for Sept.2018) but you’ll see below that it definitely makes up for the extra waiting time!

Distant GXS syncronization

The generic data distribution system that RetroShare uses for distributing forums, links, channels and emails had originally been written to work friend to friend, transmitting data between pairs of subscribed connected users. This makes it quite robust but still requires a full chain of subscribed friends to connect you to the source of some data in so that you get new posts. In version 0.6.5, we introduce an optional distant synchronization of GXS groups (e.g. a single Channel is a GXS group) that allows to sync subscribed groups using authenticated encrypted tunnels. With this technique, people can search, subscribe and synchronize groups that are not subscribed by immediate friend nodes.

This has been enabled in Channels only, as an experimental feature. If it proves useful enough we’ll extend it to forums and potentially all GXS services in the next release.

The new search at the bottom of the channel list sends a query across the network and only reports channels that *are not* already advertised by friends nodes. When a new channel is reported, right click on it and hit “request data” to retrieve the full channel group (which means encryption keys, mainly). If available, it will soon be moved into “Other channels”, from where you can subscribe to it. We may automate this a bit more in the future in order to make this 2-step process simpler, but the current setup helps us diagnose potential problems.

As a side effect, channels synchronize using tunnels only when necessary, which we figure out by looking at whether friend nodes already supply information about a particular channel or not.  This does not totally prevent “isolated islands”, and we may improve this heuristic a little bit in the future.

Forum moderation

In this release, forums can have moderators, who are allowed to edit anybody’s post. Moderators are set by the owner (meaning the creator) of the forum, as part of the forum group data that is signed by the admin key of the forum owner.

Forum moderator list

Adding/removing moderators to a forum.

Posts still cannot be deleted. The reason for this is that because posts are duplicated on all subscribers, it would give a false sense of security to let someone believe that he/she actually deleted a given post.

Post can also be pinned by the forum administrator, which makes them stay on top no matter which sorting method is chosen, with a colored background. Only the administrator has the right to do this, which allows forums without moderators to have pinned posts. It also simplifies a lot the security of it since the list of pinned posts is part of the forum group data that is signed by the administrator:

Forum post pinning example.

Pinned posts stay on the top no matter what.

Note: In order to edit a pinned post, you need to “un-pin” it first. We’ll fix that later, and it is a side effect of how post versioning is implemented below.

Forum model

We implemented a new QAbstractItemModel for forums, the beauty of which can only be seen by reading the code (sorry). Still, it makes the forum loading much faster and more memory efficient. Because this is new code and is rather complex, you may find errors. Please report them back to the dev team.

Tor only and clear-net versions merged

In version 0.6.4 we used to provide 2 different versions: one working on the clear-net and one working over Tor. Both versions are now merged in the same executable. When creating a new certificate/node the user can choose the node type as Standard node or Hidden node. A third option allows to manually configure Tor/I2P to allow all the combinations previously possible:

When a hidden node is created or started, RetroShare will look for Tor (which it should find if properly installed on your system) and automatically configure it to run a hidden service which will be used by your friends to connect to you. It will complain immediately if something’s not working in the configuration of Tor.

Of course hidden nodes can still talk to non-hidden nodes, but connections only happen from the non-hidden node which has Tor proxy configured. This is usually sufficient, and this way no Tor exit nodes are involved whatsoever. We also made sure that the discovery system *never* sends IP addresses to Tor nodes (they don’t need IP addresses but could potentially leak them between friends). The UI in Tor-only mode is overall much simpler, since UDP, Relays, DHT, Nat punching code, IPs filtering, etc. are irrelevant in this setup. As such, Tor nodes offer a much better level of connectivity.

Remember that Retroshare over Tor is a very powerful tool. Use it wisely!

Automated JSON API generation

The experience of rapidly developing a Retroshare based Android chat app with Qml plus the old libresapi convinced us that exposing a complete JSON API would be strategic for the growth of Retroshare ecosystem. Still we realized that libresapi wasn’t capable of supporting that growth because for every function of the Retroshare C++ API we wanted to expose trough it we had to manually write wrappers for it, which is a very boring and error prone job. This caused libresapi to be always severely behind libretroshare in terms of functionalities. This problem motivated us to investigate for a better approach, and led us to a radical change on how we expose the JSON API.

The new approach uses restbed as a HTTP framework and automatically generates JSON API stubs directly from Retroshare C++ API code and documentation, with the help of doxygen capability to parse the code and outputting XML description of it integrated with the inline documentation. This way any properly documented method from Retroshare C++ is now available without extra work trough the JSON API. Another benefit is that if we change a method or add a new one the change apply to JSON API too without need of extra work.

This is very important for developers who would like to develop decentralized applications without being a distributed networks or a C++ god. Now any JavaScript, or python or Lua or whatever programming language script kiddie can harness the power of RetroShare to develop it’s own custom decentralized app (and some are already doing it). We also started upgrading the existing Web interface with the new API, and hope for Google Summer of Code to make finish it during 2019.

We’re also presenting the JSON API this year at FOSDEM’2019.

IPv6 support

The long time efforts to bring IPv6 support to RetroShare have finally brought their fruits. The IPv6 support is finally stable and merged in the official version which we even managed to do without a code refactor chain reaction, and without breaking retro-compatibility with Retroshare 0.6.4.

Most connection problems in F2F networks are due to NAT being practically omnipresent in the IPv4 Internet because of address space exhaustion, punching a connection trough a NAT is a time consuming and not always a reliable task, as result developers and user experience are not always as smooth as one would desire. IPv6 instead offers plenty of addresses so every device cat get at least a public address without a NAT interfering with your connections, so connecting to your friend nodes is almost instantaneous and very reliable when IPv6 is available. If your Internet provider is IPv6 ready you will benefit of seamless direct connection to all your IPv6 enabled friends nodes, otherwise we suggest to look for an IPv6 enabled internet provider to improve your Internet participation.

Direct node-to-node chat+mail is now an option

As we’re planning to re-design the chat system entirely, we need to get rid of the double chat system (between nodes and between identities in People) that is currently very confusing to new users. We added an option to allow to hide the interactions between nodes (chat and messages) and the broadcast system between nodes (compile with CONFIG=no_direct_chat), which leaves only chat and messages between identities.

However, in order to help some users with this abrupt transition, two new features improve a lot the distribution and visibility of your non anonymous identities:

  1.  Your own signed identities are automatically advertised to friend nodes using discovery. When friend nodes do not “know” them they will request them;
  2. RetroShare allows you to automatically add identities owned by friend nodes to your contact list.

The combination of these two features makes Mail and Chat much easier for newcomers as they can immediately reach identities hosted at neighbor nodes.

Various bugs fixed

As usual, we spent days fixing lots of bugs (and probably introducing new ones as well), among which:

  • we fixed the display of single chunk transfers was wrong;
  • the display of updates in forums and circles was also fixed;
  • files are not anymore hashed while being modified (e.g. written). This is quite easy to achieve by looking at last modification time stamp and prevents files to get the wrong hash;
  • file transfer now aggressively re-requests pending chunks at the end of transfers, which prevents slow sources that have been set to a given chunk to slow down the termination of the transfer. Now the last chunks may be received twice, but they come much faster.
  • we fixed the nefarious crash in UDP connection with OpenSSL 1.1+

What’s next?

Now that we have a API to talk to libretroshare, it is time to get into serious business. There’s a bunch of very interesting possibilities of what to do with it, including a social network that would run in a browser while talking to the RetroShare background application. 

2019 will also be the year where RetroShare gets into Debian which is a condition to get in Tails as well. This is the next task we’ll work on after this release is published.

Summary

A nice release, which we hope will bring you security, privacy and more usability.  The JSON API and IPv6 integration are the work of G10h4ck.  We also would like to thanks all contributors to this year’s huge set of pull requests (Phenom, Sehraf, hubernd, RetroPooh, Chelovechisko, Zapek, San-Josep) and tests and feedback (Ghibli, ASmith, Jolavillette).

We provide builds for MacOS, Windows and most Linux distributions. More information about this on our Downloads page!

Cyril & Gio.

 

Release notes for v0.6.4

This release brings a new version of Retroshare optionally bundled with Tor, some security improvements, many GUI fixes, and an experimental Android version.

Tor-Only package

Since version 0.6.0, Retroshare is able to run over the Tor network, using a Tor hidden service to create a so-called “Hidden node” which IP is not visible even to friends of this node. The configuration of a hidden node however was not very easy, especially on windows. With version 0.6.4, we release a special version of Retroshare which bundles Tor and configures it automatically. As a result, creating a hidden node is now one click away.

How does it work? At start, RS launches Tor using a QProcess, and uses the socket-based control system of Tor to configure a hidden service. Then it configures itself to use that hidden service. To do this, we borrowed some code from the Ricochet project. After you created your node, Retro-Tor will temporarily display a status window. When the bootstrap reaches 100%, you’re good to go:

Running Retroshare over Tor has a number of definite advantages: it does not require firewall management (Tor does it for you); you do not need a DHT to find your friends (Tor does it for you), and whatever code is tied to ensuring security of your IP information is not needed anymore.  We thus removed IP filters.

We envision multiple use cases where this special “retro-tor” combination would be very useful. To list a few:

  • while Retroshare would normally require that you only create connections to some trusted friends, Retro-Tor  allows you to safely connect to strangers, who cannot know your real identity (meaning your IP address and physical location) when they connect to you. Retro-Tor therefore allows you to easily test Retroshare by exchanging certificates with stangers from the internet in a way that is perfectly safe. If you’re happy with the software, you may turn to the “classical” Retroshare to create a network with trusted friends.
  • your work or activity requires you to securely and very privately contact trusted persons, while being able to send large files, forum posts, or have a multi-party discussion? Then Retro-Tor is a valuable option.
  • you want to anonymously share information/data (i.e. without ever revealing your identity and physical location to anyone)? Then create a hidden node with Retro-Tor and exchange certificates with interested people.

In the future, depending on the feedback we receive, we may unify the Tor only and vanilla Retroshare versions into a single package that will optionally run its own Tor. This will allow users to easily connect to hidden nodes and bridge between the two networks.

It is important to note that connections over Tor may take a little while to find their way, especially the first time. But they eventually always do if your friends are indeed online. Also, in this version, Retroshare will only start after Tor manages to bootstrap correctly.

Retro-Tor is available as an AppImage on linux, a zipped archive on windows, and a DMG archive on MacOS. It can also be compiled from the master branch using “CONFIG+=retrotor”, but the packages do not need you to install Tor as the executable is also in the packages. In is referred to in the download page as “Tor Only” version. Retroshare+Tor represent a significantly powerful tool that raises anonymity to a total new level. Use is responsively!

Experimental Android build

For the first time we offer a very experimental version of Retroshare on Android. This version only allows to connect to other peers and distant-chat your friends. This however constitutes a major step toward a more complete Android version since a lot of work was done to get our code base to compile and properly run on smartphones, and to create a Qml UI for it.

Security

We reviewed the security of the signed TLS handshake and made the following improvements:

  1. the signature of certificates used to require a double hash. While not a security problem per se, this was an unfortunate design choice.
  2. PGP signatures are now only possible in SHA1, SHA256 and SHA512 format. Certificates signed with another hash algorithm will be rejected.
  3. SSL ids are now computed using a secure hash function. While not a critical security problem (since signatures are always checked), the previous method to produce SSL ids was not coming with a theoretical guaranty of how difficult it was to create a node/location pair that had the same SSL id than an existing node.

In order to make all these changes backward compatible, we adopt 2, and ship the code to handle 1+3 in version 0.6.4. In the next version, we’ll switch the default hashing algorithm to SHA256 and switch the way SSL ids are formed. Switches for these changes are at the end of retroshare.pri

Finally, hidden nodes do not accept nor pass any IP information through discovery. That means in particular that a hidden node that is friend to 2 normal nodes that trust each others will not transmit each other’s IPs. In addition to a security improvement, we think it is better that hidden nodes do not store any IP whatsoever since they will not use it anyway.

Improvements in GUI

We performed the following improvements on the GUI. To name a few:

  • All downloads now use the “RsCollection” Dialog box, which allows to choose a destination directory, de-select some files, etc;
  • links can encode (small) hierarchies of directories;
  • file lists now are more efficient, and have a search function in tree mode;
  • the keyring and download lists are now handled by a QAbstractItemModel, which makes them much more efficient when displaying many entries;
  • chat lobbies are renamed into “chat rooms”;
  • we added two kinds of links: chat room links, and identity links;
  • log remembers the new forums/channels, and will not display them again when you restart.

Backend

  • we added a cache in GxsGrpMeta (but not GxsMsgMeta), which improves a lot the speed of accessing group lists.
  • we fixed a few bugs in channel permission sharing, and improved the update of GUI when new groups are published.

What’s next?

We have a good roadmap for 2018. Here’s some bits:

  • If Tor+RS works fine, we may embed Tor in normal nodes as well so that people can easily create bridges between the Tor and non Tor networks;
  • we may have plans to get rid of OpenPGP-SDK, which is a non elegant piece of C code, and may use elliptic curve cryptography in our certificates in replacement to RSA;
  • in its current state, the serialization code only allows to serialize to binary format. We will extend it to handle JSON serialization, and if possible create an automated wrapper for API functions;
  • IPv6 support;
  • create a friend suggestion system to work in Retro-Tor. This will be a possible GSoC project;
  • open plugin system to GXS;
  • distant sync of GXS groups via tunnels
  • re-implement the chat layer;
  • re-design the network layer which has become a bit messy.

We need lots of help to get all these things done. If you want to help you’re welcome to do so. In particular, this year again, Retroshare will participate in Google Summer of Code, for which the topics are available on Freifunk’s projects page.

As usual, many thanks to all “non official” contributors to this great project, namely Phenom, sss for their pull requests, ghibli and ASmith  for feedback and testing, anmo for experimental packaging, and all others who helped in any way. Finally we express our infinite gratitude to the autors of the Ricochet project for their code to control Tor using sockets.

Packages: https://github.com/RetroShare/RetroShare/releases/tag/v0.6.4

Download page: http://retroshare.net/downloads.html