Category: Quality

Photographic Evidence is Dead

Fake Image, Fake NEWS, Fake Trust.

-----BEGIN TEXT-----

We have now witnessed the death of almost 200 years of photographic (and other recorded) evidence. Images, videos, and audio recordings can now be easily faked or altered in ways that cannot be detected. Digital technology has made this happen. Analog media is continuous, so subtle modifications can be noticed. However, digital media has discrete bits that are not dependent on the bits around them.

It is time to relearn what was so obvious to our ancestors: the SOURCE is more important than the content. “Do you trust or believe the source?” This can be a personal choice, but we no longer have the convenience of “socially accepted” sources.

Sources

  • The mainstream media can not be trusted without question; they are polluted with greed.
  • The government cannot be trusted without checks; they are polluted with keeping control.
  • Non-mainstream sources can not be trusted without question; they are polluted with the desire for control.
  • People on social media sites can not be trusted; many repost fake news to get the algorithm’s attention.
  • Individuals cannot be trusted until you have seen or heard their patterns of bias.

Trusted Sources = Reputation

–MORE–

The old patterns of trusted sources have to be rebuilt. Luckily, we have some technological advantages that didn’t exist 200 years ago: public/private key encryption makes it possible to build “webs of trust” that can be wider than your contacts. The bits of any digital files (text, images, videos, etc.) can be signed by individuals, so you can trust that those exact bits have not been tampered with, after that person has signed them. Also, the chain of trust to that individual can be followed to help determine if that person can be trusted to have signed an untampered file. Yes, this can be complex, but it can also be simplified if groups want to build a reputation for being trustworthy by allowing audits and third-party witnesses who can verify that the images (or other evidence) matches what they have seen.

This is a new age: all sources will always be questionable, and all the old sources must build up their reputations from ground zero with a verifiable audit trail. For journalists, the audit trail of sources could be kept private but still be auditable. People smarter than me know how to do this. In this new age, putting your trust in an unverified source is foolish. “We are all being deceived.”

AI Fakes

Congress wants to pass laws requiring AI-generated content to be labeled. Okay, but content publishers must also attest that the content was NOT AI-generated or that an original version was or was not modified. If they have lied about the content, they should be identified publically, and they could be charged with making liable comments. If some content was modified, the original sources must be provided somewhere, with a signed audit trail. It does not matter if AI or humans created the content; what matters is whether the person or organization is telling the truth.

My message to big corporations and big media companies is that if you are unwilling to provide auditable chains of trust, for your content, we should assume your content is untrustworthy.

Who to trust?

Currently, it comes down to a personal choice of who you will believe? Questioning the sources should always be acceptable. It would be rude to not allow others to question or to not see a verifiable “chain of trust.” But even then, the chain will still end with individuals or organizations; do you believe they can be “trusted”? Reputation will become a precious commodity, as it was in the past.

My touchstone for evaluating someone else’s level of trust is that if they do not question their sources, I will rank them lower than people who question even “reliable” sources.

Some Technology Help

These technologies can “help” build trust, but they can all be compromised. We should never again put unconditional trust in any medium.

PGP (Pretty Good Privacy) was initially released in 1991. Secure encryption and signing have been around for over 30 years! It is long past time for all web pages, documents, images, and even videos to be dated and signed with secure and auditable signatures.

Here are some newer techniques that could incorporate signing with chains of trust.

  • Fediverse – supporting decentralized applications
  • ActivityPub – a protocol for decentralized applications
  • Ghost – micro blogging
  • Mastodon – decentralized social network
  • Matrix.org – decentralized encrypted social network
  • PeerTube – decentralized

Companies and individuals must start building reputations that we can trust and verify. If we don’t, everything will be “fake,” and democracy will falter.

A Crude Example

This section is for engineers. Non-engineers can skip the rest of this article.

I’ll show a crude example using the GNU Privacy Guard tool (GnuPG, gpg), which implements PGP encryption/decryption and signing with public/private keys. This is a “crude” implementation because it can only be used by a few people who know how to use CLI (Command Line Interface) tools. Wrapping this code with a GUI (Graphical Interface) would be a much better implementation. Or even better, embedded keys and signing into a web platform so it is mostly hidden.

All of the code and sample files used in this article can be found in this GitHub repository: example/photographic-evidence-is-dead

gpg can sign a file.

  • If the file is a text file, the signature can be appended to the file.
  • If the file is binary, the signature can be output as a separate file (detached).
  • A separate file is also useful for text files because the signature is 14 lines of nonsense text.
  • The signature could be converted to a QR image.

Signing a text file

These gpg commands will sign and verify a signed text file.

Sign:
gpg --default-key test@example.com --clear-sign -o sample-1.txt.sig sample-1.txt

Verify:
gpg FILE.txt.sig

I wrote a script that makes it easier sign and verify files with gpg. The script can be found here: gpg-sign.sh The usage help can be found here: gpg-sign.sh.md

Before gpg can be used for signing, you need to create a key. Here is a quick way to create a key; just hit Enter to accept all the defaults. Usually you will use an email for your key’s Id. (Note: gpg will create the ~/.gnupg directory for its files.)

gpg --full-generate-key

To demonstrate the script, we need a text file. Here is an example input file we will sign. sample-1.txt

Gettysburg Address

Four score and seven years ago our fathers brought forth on this
continent, a new nation, conceived in Liberty, and dedicated to the
proposition that all men are created equal.

Source: https://en.wikipedia.org/wiki/Gettysburg_Address

Sign sample-1.txt with the gpg-sign.sh script using the “-c” option to create a signed “clear-text” file, with the signature appended to the file. When this is run you will be prompted for the test@example.com passphrase.

$ gpg-sign.sh -c -k test@example.com -f sample-1.txt

sample-1.txt.sig is the signed file. (sample-1.txt is not modified.)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Gettysburg Address

Four score and seven years ago our fathers brought forth on this
continent, a new nation, conceived in Liberty, and dedicated to the
proposition that all men are created equal.

Source: https://en.wikipedia.org/wiki/Gettysburg_Address
-----BEGIN PGP SIGNATURE-----

iQHFBAEBCgAvFiEEPuEu+RQEXLLLgaza4jlYZSPGd3MFAmdgisARHHRlc3RAZXhh
bXBsZS5jb20ACgkQ4jlYZSPGd3P6Swv9Ezs0uvkNiStJBs4QWWrv1p1y4icTmgOX
6u0c9H4750sfSll4SQ30I5j8xC9W28TyEHGKj9QOaTwK5kwOs903W3EmA7S8g1Bv
gi3V3LGXXTeAJfhTcDPnMmTLwrpkDSHaSGWH+etPtG2vM77X89s3D1oNFKeGrBER
2P8BfPQlK3hMQrFy4trwU8Jr94Pg8/B3/A3ex8xDCuo5HpDgMYymaM/JFn3CIju+
VQX3fnF++p9+rb9MXAqEEEDsOgxH1JxrDgLuPOMadVwLy+GvDDje+h54lMYiSf9e
Wg/ZsTPTXF5f8fOMfLpDCnbY2jhYZ7MOUPfwtz4Tz9MXhDqKWzc7rRmRNt+dWQMB
JVohMrQKcDE4lp5vTN/bDy6Imzi5HQPgWxvAUjWYNXHnpwjnb00h7sqYI2XdVv32
HD5a20Dkw7aOqMjoAGt1MDKNNELvZxqU+rrC374tkW7yB8zoZKKtVtO2wOk9vvBA
8FpDNz4bD8ApTNfhaAyKQuJT0LjJUgGl
=dHAM
-----END PGP SIGNATURE-----

Signing a web page

Problem: the signed content cannot be even one bit different. This works well for program files or for text files that are emailed, but this is a problem for web pages because many websites embed different tags in the page dynamically (usually with JavaScript), and the format can be changed at any time.

Here is an html example sample-2.html. (note: spaces are put after ‘<‘ to make the tags inactive on this page.)

< !doctype html>
< html xmlns="http://www.w3.org/1999/xhtml">
< head>
  < meta http-equiv="content-type"
        content="text/html; charset=utf-8" />
  < title>gettysburg address< /title>
< /head>
< body>
  < h1 id="gettysburg-address">gettysburg address< /h1>
  < p>---begin text---< /p>
  < p>four score and seven years ago our fathers brought forth on this
  continent, a new nation, conceived in liberty, and dedicated to the
  proposition that all men are created equal.< /p>
  < p>source: < a href=
  "https://en.wikipedia.org/wiki/gettysburg_address">gettysburg
  address< /a>< /p>
  < p>---end text ---< /p>
< /body>
< /html>

One solution is to normalize the files so that only text is signed.

  • Cut/paste a defined range of text.
  • Remove all tags and collapse all white space and line breaks to one space.
  • Sign the normalized file, with a detached signature file.

The routine that normalizes the text would be shared with the signature so that when a user cut/pastes the text from a web page, the normalized text would match the text that was signed.

I wrote a script that will, remove all tags, remove extra spaces (and new lines), and it will leave html links. the script can be found here: just-words.pl

Normalize sample-2.html to sample-2.txt with just-words.pl

$ just-words.pl < sample-2.html > sample-2.txt

sample-2.txt file. (In the file, there are no line breaks.)

Gettysburg Address Four score and seven years ago our fathers
brought forth on this continent, a new nation, conceived in Liberty,
and dedicated to the proposition that all men are created
equal. Source: https://en.wikipedia.org/wiki/Gettysburg_Address
Gettysburg Address

Now, sample-2.txt can be signed, with the signature output in a separate file.

$ gpg-sign.sh -f sample-2.txt -s -k test@example.com

sample-2.txt.sig is created.

-----BEGIN PGP SIGNATURE-----

iQHFBAABCgAvFiEEPuEu+RQEXLLLgaza4jlYZSPGd3MFAmdgjNsRHHRlc3RAZXhh
bXBsZS5jb20ACgkQ4jlYZSPGd3PntAv+Mwp7wbnhGcC+HTBZnvqLOB0J1qa0m9zj
FGxt+rMKoB2SuZ2cILghqA4n3SGAtZj8HgfHA2OqTuRTi2Vnvv3ZIRxv0gDu9jNR
+ruOq41WMyWhEERZHt1m8XUHgrDoOJZjAPe7ocgZpN90IiEUGPlkbUApAXzOBj2v
S/eC7T/68lj+gZng5XOsZF20PG9KoXvOA1sbWxmbXUM6F6t4FpNbOKXj/PT8DX2y
m9PM9uqTLTw0MdD+1nX7vIASqNA8Wz0nre/G6imAjTMRxvbZQ0eEDSKykptaROZD
HZCn+WpCA89tGPZL7L0GuRstDEyl6C/p9d2VoLcZX3hvi+UrVlUSz7eSLoxMfDOJ
U5u7G97PkMeLZbBnI6FfGdUVJ0sw7kMSdfNi/cdmgVaPKIYtBl0z4kqlLK/Orb+8
HJnYaBLCsr720C3wkfoeONZyeAw+DjJsS8MYW55f3yxzeC7RJMItOSuyIlPoHHH/
Ks+/eH7uJoO/zUnIlUVf82qKTiwFvGSp
=aoIe
-----END PGP SIGNATURE-----

Now the contents of sample-2.txt.sig can be put at the end of the sample-2.html file after the “—END TEXT —“. Or to make it look better a QR code image could be appended, where the QR code would be the signature text. The https://gchq.github.io/CyberChef/ is a simple QR generator. Just put all the sample-2.txt.sig text in the Input field.

For example see sample-3.html

< !DOCTYPE html>
< html xmlns="http://www.w3.org/1999/xhtml">
< head>
  < meta http-equiv="Content-Type"
        content="text/html; charset=utf-8" />
  < title>Gettysburg Address< /title>
< /head>
< body>
  < h1 id="gettysburg-address">Gettysburg Address< /h1>
  < p>---BEGIN TEXT---< /p>
  < p>Four score and seven years ago our fathers brought forth on this
  continent, a new nation, conceived in Liberty, and dedicated to the
  proposition that all men are created equal.< /p>
  < p>Source: < a href=
  "https://en.wikipedia.org/wiki/Gettysburg_Address">Gettysburg
  Address< /a>< /p>
  < p>---END TEXT ---< /p>
  < p>Signature< /p>
  < img src="sample-2.html.sig.png"/>
< /body>
< /html>

My Keys

You can find my php keys at: my-pgp-keys or (Archive)

(Image by Rob Oo from NL on Wikimedia Commons)

-----END TEXT-----

Signature

To verify this article’s text:

  • Save the page as file.html
  • Run: just-words.txt <file.html >file.txt
  • Copy the PGP SIGNATURE text below to a file. For example: file.txt.sig
  • Import my turtle.engr.pub key to your gpg (one-time action)
  • Run: gpg-sign.sh -f file.txt
-----BEGIN PGP SIGNATURE-----

iQHKBAABCAA0FiEEcBGX4HDGbbvKA+CcS98j3zUz2P0FAmdh5hMWHHR1cnRsZS5l
bmdyQGdtYWlsLmNvbQAKCRBL3yPfNTPY/am/DACyPLvCs8B3A0Ne6E+VQY6Hskgn
inuQ2WVzanAadNgg7puFaXnEMySCSHvDAO9y69NdzMXSc1yUc4NeTaZ2UsYqZmOB
vyn3QdbzmLtIuhX0NeLJgwTCTIsV4rWGsIXJMOhHW+8uV633ftX18IrfuqbPv0Su
aGXkF9/jihCl7V0uReLuA0EsxI10/mMiTpZiGCr19oKFr2DBMmUb7L7tbYmu/K+M
pYYvzBHqQrYh8KoOB90sSXC9d8wH5rv30eqitSN1zvvVYdiubWRE7o3jFjT4Ds1T
jMCuG9OGkhIDyLyP/8TwiUlmrDSXGjgDFdwbiBJ/qgTes08x4AC4YQS62THgg/VF
fORTiAY5zj4qgHKChhAt+FVCjm6SVSRVD0Qg66gtq6EBwpzPUmVHR5NjV2j+BYRW
DxidRA4aWvPVbTJZRclA9db2Aj8AhzEPNXbnag0AghZr6yVeFJeAvN7kM47xOyRI
anb3SeiiCnt13FdquTAYqhw1Le9La3XZRfhBg5o=
=gkiE
-----END PGP SIGNATURE-----

Clean Code

I’ve been learning Google Apps Script, which is basically JavaScript. I’ve also been learning Test Driven Development. Sure, let’s learn three new things at the same time!

I was motivated to try TDD by this video:

In this video, he demonstrates a TDD Kata for creating a function that will find prime factors of numbers. I want to write code like that!

(All of the example code mentioned in this article can be found at: example/clean-code.)

My google drive was getting cluttered with files that were named by others, who had a (bad) habit of using spaces and lots of other special characters in the file names. (Usually, this is because people try to encode too much information into a file name. But that is another topic.) I would download, version, and process some files, then upload them. Spaces and special characters in the file names messed up my ability to write “simple” bash scripts. So how about a renaming tool to normalize the file names? I.e., convert all the non-alphanumeric characters to ‘_’ (also allow ‘.’ and ‘-‘).

For the tests, I needed to create and recreate test folders and files in Google Drive. It was really tedious to do manually. So let’s automate it. I wrote a routine that would parse a nested array structure that represented the folders and files.

This was the object that I created. (This is the first version. I now have a version where you can optionally pass the array as a parameter.) See: test-refactor-before.js

Continue reading Clean Code

Planned Obsolescence in Software

198103463_8944be366a_z

“Fast, Cheap, Good – pick two!”

In any software project (or probably any project where you are creating something), you cannot lock down all three of these dimensions–i.e., you cannot define what you want for all of them. If you are very experienced, realistic, and have control of all external factors, you might get close. For most projects to succeed, one of these dimensions needs to be more relaxed than the others.

Fast and Good

The usual expectation is that “fast and good” won’t be cheap. Lots of people or money will be needed. But lots of people does not mean it will be done fast (see the book “The Mythical Man-Month”). So part of the solution is usually bought instead of created. That gets it done fast. But will it really be good? Maybe if you spend a lot of money, to buy something really good–essentially, you’re paying for the time that others spent on it at another company. This also assumes the part you buy will be exactly what you want.

Good and Cheap

The expectation is that a “good and cheap” solution will take a long time–that one person working on a project for a long time will be cheaper than lots of people. But will it really be good? Usually, it takes more than one person to make things good. For example reviews, proofreading, and brainstorming requires at least two people. So a few people over a long time could make a good product, but that doesn’t sound like it will be all that cheap (unless the people work for free).

Fast and Cheap

“Fast and cheap” is not likely to be good. And that is the usual way projects are started. The project gets accepted by most companies when those paying for it are convinced that it can be done fast and cheap. The assumption is that it will need to be just good enough for customers to want to buy it.

Cost, Features, Quality

But let’s break the discussion across slightly different dimensions, because “fast-cheap-good” may be a fun quip, but most product and project discussions revolve around Cost, Features, and Quality.

Continue reading Planned Obsolescence in Software

What is this blog about?

See: About