As a concrete example, assume something like this:
class Base
{
    ...
    int some_int;
};
class Derived: public Base
{
    ...
    vector<int> some_vec; // the presence of this field makes `Derived` 
                          // no longer trivially destructible.
};
Now if we do this (either directly as shown or indirectly through a smart pointer, e.g.):
Base* base = new Derived;
...
delete base;
... then some_vec will not be destroyed unless Base defines a virtual destructor. After all, given only a base pointer, the system lacks the information at runtime to know anything about Derived unless there's a virtual table that points to its functions, allowing a dynamic dispatch to occur to get to information/functionality specific to Derived like its specific requirements for proper destruction.
To delete an object through a base pointer is polymorphic behavior, and to do it safely and correctly without running into undefined behavior requires the base class to define a virtual destructor.
The question ultimately boils down to whether you want to delete objects through a base pointer. If you do, then define a public virtual destructor:
class BaseSafeDelete
{
public:
    // Provides safe destruction through a base pointer.
    virtual ~BaseSafeDelete() {}
};
If not, define a protected nonvirtual destructor:
class BaseNoDelete
{
protected:
    // Prevents deleting outright through a base pointer.
    /*nonvirtual*/ ~BaseNoDelete() {}
};
And most importantly, inheritance is something to be designed and decided upfront when designing a base class. It's not suitable as an afterthought to take a class which was not designed for inheritance whatsoever (lacking either a protected nonvirtual dtor or a public virtual dtor as a first sign) and attempt to extend it. For that, see the Composite Reuse Principle.
     
    
typeid!