Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Topics - tschak909

Pages: [1] 2 3 ... 12
1
Users / Added: Support for creating ZFS storage pools/filesystems.
« on: May 21, 2017, 04:47:38 am »
Hi all,

I've recently added support for Oracle's ZFS filesystem to LinuxMCE. The support is integrated in the same place as the RAID setup in the web admin, and is selected as another type of RAID template. ZFS offers a wide range of benefits over the traditional md software RAID that has been present with LinuxMCE since the beginning, namely:

* The underlying block storage system and filesystem know about each other, and keep track of what's actually stored. This means, no lengthy array reconstruction times!
* The filesystem natively supports compression to increase disk space.
* The filesystem supports snapshotting for backing up, restoring, or transferring pool contents.
* For RAID-Z levels, the underlying parity information is kept with each block, rather than with each stripe, avoiding the RAID pairty hole problem.
* In addition to modes that correspond to RAID-0 (pooling), RAID-1 (mirroring), and RAID-5 (RAID-Z1), there are modes to provide a double parity RAID (akin to RAID-6), and triple parity RAID (akin to what would be in essence, RAID-7)
* The ability to mix and match all of these modes, together, in various configurations to create storage pools that best suit performance and reliability requirements.

It's not all roses, there are some drawbacks, most notably fragmentation due to the use of copy-on-write for everything, but ZFS mitigates this using very clever algorithms, so in practice with media storage, this should be of minor consequence. I will provide a way to defragment the resulting pools via the web admin, when I can.

As before with traditional RAID, you select your disks, and select Create Array to create the resulting storage pools. Pluto Storage Devices will immediately appropriate the storage, and mark it for use (if use automatically is selected).

What versions?
-----------------
Currently, Trusty (14.04), and Xenial (16.04) and later have this functionality. I had done the necessary work for Precise (12.04) but for some reason, it's not installing correctly (due to a sysv-rc version mismatch, which is VERY strange!), so the linked dependency to the zfs tools from the lmce-core package was removed until I can determine what the problem is.


The following ZFS Pool types are available:

* ZFS Pool - Akin to RAID-0, but with blocks dynamically striped across the device for better overall performance.
* ZFS Mirror - RAID-1 like functionality, again with dynamic striping
* ZFS Raid Z1 - Single Parity RAID (RAID-5 like)
* ZFS Raid Z2 - Double Parity RAID (RAID-6 like)
* ZFS Raid Z3 - Triple Parity RAID (RAID-7 like)

What's done:

* Array creation (with or without hot spares)
* Deletion
* integration into pluto storage devices

What's to be done:

* Array growing (there will be no deletion of members from pools, as this isn't possible)
* Hybrid pools (mixtures of the above storage pools, needs a lot of new UI in the web admin)
* snapshotting (need to think of a proper way to present this to the user)
* Better status reporting (hooking further into zed)

Getting it:

New installs of 14.04 and later include the necessary ZFS support. If you are already running 14.04, then an apt-get upgrade should pull in the new ZFS packages, as well as the new device templates for the ZFS storage pools. After this point, you should be able to create new storage pools.

A note:

You may want to wait until posde pushes my latest updates (as of 2017-05-21), before trying, as I did a LOT of bugfixing over the last day. I will tell in this thread when those changes have been pushed into the repo.

Enjoy,
-Thom

2
Developers / VLC Player status - multi-room sync checked in
« on: March 27, 2017, 09:42:01 am »
It's almost there. While video looks to be very well in sync, I am not recording the audio because there is approximately a 40 to 80 millisecond difference between media directors, which must still be addressed.. possibly through a processing delay... But as you can see, it works, even with DVD media (and menus!)

With this feature, this means that VLC Player is ready for beta testing.

https://www.youtube.com/watch?v=pQKCSXCGRSg&feature=youtu.be


3
Users / biosdevname causes issues during diskless boot.
« on: February 26, 2017, 01:26:30 am »
A ticket for this is here: https://git.linuxmce.org/linuxmce/linuxmce/issues/2733

When linuxmce is installed via trusty-i386 server, biosdevname is installed, which is a set of initrd scripts that attempt to name various devices, including ethernet devices, by their relative hardware position as reported by the bios:
The facility is described here:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Consistent_Network_Device_Naming_Using_biosdevname.html
In our case, any embedded (LOM) ethernet devices get called emX instead of ethX, which blows up our initrd, and causes a kernel panic shortly after IP-Config successfully completes.

What I suggest, is a two pronged approach to fixing our initrd:

(1) remove biosdevname during lmce install, with a corresponding initrd update. for the short term
(2) do more testing and auditing of biosdevname to see how consistent the naming is, and adapt our scripts accordingly, for the long term.

4
Users / Diskless Workstation installation boot and ACPI=on or off
« on: February 25, 2017, 10:06:36 am »
Hello everyone,

acpi=off was added to the Diskless default boot, during the early days of Pluto 2.0, because of hardware that contained faulty DSDT tables, which did not properly initialize the I/O controller hardware. Since the BIOS in these early machines (circa 2004-2007) initialized the I/O controllers and embedded hardware to a point where the linux kernel could work around the bugs, it was a way to get the non-working machines to boot properly.

With the decreased use of legacy BIOS in the x86 world, and the emphasis on decreasing POST and boot times, firmware engineers moved the initialization of critical I/O subsystems to the operating system, utilizing ACPI to discover, enumerate, and provide the needed data to bring up the I/O controllers and other embedded devices. If ACPI is not turned on, these devices are typically in a non-working state, causing devices to not be discovered, to kernel panics while the kernel brings itself up.

It would be beneficial to hear from everyone, as to the effects of turning this attribute on or off while booting the diskless workstation, does it stop kernel panics? do the NICs initialize properly?

There is a ticket open for this issue:
https://git.linuxmce.org/linuxmce/linuxmce/issues/2731

Please let us know,
-Thom

5
Developers / Fixing External Media Identifier
« on: February 13, 2017, 08:16:42 pm »
Hello everyone,

I've been on hiatus for the last year and a half (actually a bit longer), as I needed to take care of other things in my life...but am back now, and squashing bugs on my list.

One of the things I had left in an unfinished state was debugging the reverse engineered External Media Identifier, which identifies CDs and DVDs inserted into the system and tags them appropriately.

Due to (what is now known as) an unusual compiler bug dealing with 64-bit pointer alignment, I had to disable the DVD identification code that I had written which replicated the behavior found in the original External Media Identifier.

This code now works, and I've been able to identify a series of both DVD and CD disks correctly, all of which have corresponding entries in the AMG database.

I need a few people who have DVD and CD collections to ping me, so I can give an External Media Identifier binary to test, and report issues, so that I can finally get this piece dealt with.

-Thom

6
Users / Game Player builds on Hiatus
« on: January 13, 2016, 08:46:15 am »
Hi guys,

Sorry, for various reasons, I've had to ask phenigma to remove the game player from LinuxMCE.

This is temporary, and there are many reasons behind this, all of them my fault, and the decision was mine alone.

I am taking the opportunity to do several things:

* get rid of the 3 GIGABYTE deb packages containing metadata. Replace them with a web service that serves the relevant metadata.
* re-work how metadata is stored and related, so that it can scale much better than what we have.
* provide a way for user overrides without explicitly needing to replace the attributes in web admin.
* provide a way to control the emulators in a consistent manner. This is exceptionally difficult, and systems like Hyperspin have the benefit of a MANDATED OPERATING SYSTEM LEVEL MESSAGING SYSTEM to pass events to the emulators, without requiring patching. Being Linux, we have no such luxury, so I have to do some SERIOUS experimentation and research to determine the best way to control the emulators
* pull the build of the emulators away from the build system, leading to cutting the build time of LinuxMCE in half, at least. (Yes, my stuff was adding hours to the build process!)
* Dockerize the build system to allow for the latest versions of MAME to be built with precisely the dependencies needed, as well as providing a modular way to keep the build up to date.
* Provide a way to select which version of MAME to use. For those of us with varying collections, this is critical.
* Use hashes to identify media and media attributes.

It will all be back, I just need to do some serious re-working of it, as years of experience have shown that I implemented it wrong in the first place. That's okay, it's all a learning experience. I know better, now.

What are your thoughts?

-Thom

7
Users / Is anyone using the Game Player?
« on: February 15, 2014, 06:15:26 am »
I'm just curious.

-Thom

8
Developers / Improving Performance of Play All
« on: September 30, 2013, 02:51:04 am »
Hey guys, Looking for some database help here.

Basically, if you have a LARGE media library, say, 20,000 pieces of audio, and you hit Play All to play them all, there is a considerable amount of processing time that happens, before anything appears to happen from the user's perspective.

This is caused by this snippet of code:

http://pastebin.com/LqWuDeDy

(all discussion here focuses on the pluto_media database)

What's happening here, is:

* When a user hits Play All, it sends an MH Play Media command to the Media Plugin, the filename is !Gxxx, e.g. !G250, that is, the current mediafile grid for Orbiter #250.
* The code snippet catches the special !G case, and immediately embarks upon a loop to traverse through the table, grabbing the cell values.
* For each cell you see, there is the text you see, and the actual value passed back to the code, and depending on which view you see, you can either be looking at a pile of Attributes (Director, Genre, etc.), or at a pile of files (the Title view, Filename view), so you'll see e.g. a set of !F1234, !F1235, or !A1234, !A1235, for PK_File 1234, 1235 or PK_Attribute 1234,1235 respectively.
* Then there are the two if's... for files, our job is easy, we get the appropriate row from the File Table, and put together the directory and file path, and then pass that along.
* For Attributes, this is more involved. We need to get the Attribute, then we need to match that against entries in the File_Attribute table, and grab all the matching PK_Files for each PK_Attribute, and THEN do what we mention above.
* At this point, the returned full path to the file is then passed into a new MediaFile object, and then added to the deque of MediaFile objects to become the playlist.
* Then the process repeats, until ALL of the cells have been traversed.

This process involves a potentially large number of individual queries the size of toothpicks. These toothpicks are small, and are inconsequential for the database to query and return, but, we also have the process of taking that data, peeling it apart, and acting upon it by doing more tiny queries, in a loop, all of this happening serially, and therefore incredibly inefficiently.

Why does this matter?

Well, besides the obvious effect of longer and longer delays from the user's standpoint, any code built on DCE has a built-in deadlock prevention timer. That is, if you send a message, and that message takes longer than 30 seconds to complete, DCE will assume that things have stalled, and the offending process will be terminated. Since Plugins like the Media Plugin run in exactly the same address space as DCERouter, that means the DCERouter gets killed if some work in a plugin takes too long to complete! Router goes "bye-bye;" whole system says, "sayonara."

So even if you can PUT UP with the egregious delay while asking the system to play a large number of media, there will hit a point where your system will take too long to process the "Play All", and your system will appear to crash. First, your Orbiters, then the core. It will recover, but your intended goal of playing the media won't happen.

Ok, so, what's the goal?

We need to make this block of code more efficient, and the way I can see that happening, is to make a proper database query to return ALL of the PK_File entries that match either the list of !F or !A entries that are present in the currently visible datagrid. This can subsequently be peeled apart linearly into MediaFiles and dequed appropriately. I believe this will decrease the delay to less than a few seconds, if not less than 1 second, for 99% of cases,

but, I am not an SQL wizard, and need some help to try and construct this query.

Any thoughts?

-Thom

9
Users / Atari 5200 and Intellivision now have Keypad overlays!
« on: September 05, 2013, 05:28:46 am »
The upcoming 1204 release will feature a Game Player with Keypad overlay support for Atari 5200 and Intellivision games, which _really_ need these keypad overlays to play the games.

Here are some Photos, enjoy :)

https://sphotos-b-dfw.xx.fbcdn.net/hphotos-ash3/1272106_10151550939417115_1965740363_o.jpg <-- Atari 5200 / Star Raiders

https://fbcdn-sphotos-f-a.akamaihd.net/hphotos-ak-ash4/1271492_10151553245637115_557938787_o.jpg <-- PBA Bowling / Intellivision

https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn2/1172552_10151553245597115_959071502_o.jpg <-- Armor Battle / Intellivision

https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-ash4/1273384_10151553245602115_926171992_o.jpg <-- Auto Racing / Intellivision

https://sphotos-a-dfw.xx.fbcdn.net/hphotos-frc3/1167519_10151553245742115_1874943942_o.jpg <-- Boxing / Intellivision

https://sphotos-a-dfw.xx.fbcdn.net/hphotos-prn2/1262558_10151553245732115_935462222_o.jpg <-- NBA Basketball / Intellivision

And an earlier one, showing MLB Baseball:
https://fbcdn-sphotos-a-a.akamaihd.net/hphotos-ak-frc1/812655_10151549830647115_592893759_o.jpg

As you can see, these overlays are integral to the game, and are needed to play the game in the first place, so they've now been added.

-Thom

10
Users / Kickstarter Campaign to fund a programmer?
« on: September 04, 2013, 07:15:56 pm »
Hi.

I am starting this thread to gauge interest, and promote discussion.

If I were to start a KickStarter campaign, to fund $50,000, for me to work wholly on LinuxMCE for a year, would there be enough interest? You know me. You know what I can do. What are your thoughts?

-Thom

11
Users / Weather Plugin Developer needs data entry help!
« on: August 24, 2013, 03:16:55 am »
Hello everyone,

I need your help.

In the pluto_main database, there are three tables, Country, Region, and City, which are ultimately used to find a particular city or town. I want the Weather device to use this data, to pick not only which weather data to display, but also which weather radar data to automatically select for display.

But for this to work, I need lots of data mapped.

Here is a list of all the maps that Weather.com offers from its map room.

https://96.226.245.101/work/maps.xml

Some of them are animated, some of them are static.

And these need to be mapped to countries and regions in the LinuxMCE database, so that when the Weather device looks at your installation, it will select the most appropriate maps to display.

Does anyone want to help? Yes, this is exceedingly tedious, and potentially thousands of mappings may need to be done, but in the end, it will allow for an ease of use and a "just works" functionality that everyone can appreciate.

I will point out, that not EVERY single city needs to be mapped, just the countries and regions.

I would be very indebted to those who help out, and help to make the weather functionality in LinuxMCE heads and shoulders above everything else out there!

Thanks for reading,
-Thom


12
Developers / Revamped the Voicemail UI on Orbiter
« on: August 23, 2013, 04:04:20 am »
I am currently checking in a multitude of changes to Telecom_Plugin and to Orbiter, to facilitate a better voicemail screen.

(to forum members: No, I will not back port this to 10.04. This is for the next release. Please don't ask.)

Current features:

* Able to distinguish between new and old voicemails both with textual label and color coding.
* Able to play individual voicemails. Voicemails are moved from new to old while playing, and UI is updated.
* Able to delete individual voicemails, with confirmation. Any deletions also update on other orbiters which may be viewing the same screen.
* Able to skip forward/backward 2 seconds in voicemail.
* Voicemail playback uses resume feature, which for now causes the remote not to display while playing back.
* Able to delete all voicemails for user, with confirmation.

To do:
* Play All. This is actually implemented, and is checked in, however, I have HIDDEN the button for now, because MH Play Media's resume functionality is broken, please see below.
* Caller ID. For some reason, Caller ID is NOT making it across to my individual voicemail metadata text files. It was the last time I did any work on this, and I haven't a clue why. :(
* Squeezebox/Squeezeslave playback. Figure out why this isn't working. Maybe a file access issue.

A note about MH Play Media resume:

MH Play Media has a resume parameter. This was defined by Pluto so that something else could jump in and temporarily replace a playing stream, and when it was done, to return to the previously playing stream (if one existed). It also has the benefit of not changing the active remotes in the room, so as to not confuse anyone.

However, this functionality is currently very broken.

I believe the playback events (playback started/finished) are not properly setting state, and this is causing some weirdness, namely that I can only play one thing in a stream, even if I have multiple MediaFiles in the deque, I only hear the first one, before it drops, and when the stream is finished, the previous stream is not continuing to play. I am trying to track this problem down, as a great number of useful features can be derived from this one flag, which we desperately need, but finding the source of the breakage is not proving to be easy; may take a while for me to solve this, so for now, the Play All button is hidden.

All of the relevant bits are being checked in right now, and will be placed on the list of stuff to build shortly.

-Thom

(posted simultaneously to telecom working group, and to forum)

13
Users / Coming Soon: Weather view, and Weather Events
« on: August 15, 2013, 12:04:54 am »
Hello everyone,

If you've been following the Developers Forum, you may notice that I am developing support for Weather views in Orbiter, and Weather Events, for the next release.

There are two parts being developed, the first part is the Plugin, which runs on the router, and holds the weather information gathered by various weather events (Temperature changed, forecast changed, etc.), and updates the Orbiter displays appropriately,

The other part concerns the Weather devices, which would gather weather from a source(s) and emit events for the Weather PlugIn to catch. This data could come from a weather station, or from online sources such as AccuWeather, The Weather Channel, etc.

Here is a preview video showing the plugin part working. I am feeding the Weather Plugin some fake data, courtesy of MessageSend, so I can test everything.

http://www.youtube.com/watch?v=ofw-yYRRbts

More information coming soon, enjoy!
-Thom

14
Users / At Home with LinuxMCE: Photos
« on: July 26, 2013, 09:37:06 pm »
Hello everyone,

Starting a new thread for pictures from users, showing living at home with a LinuxMCE installation, have fun! :)

I'll start with my latest pics, playing Tetris in the Living Room.





-Thom

15
Users / New Video: LinuxMCE and the Joggler
« on: July 26, 2013, 06:28:37 pm »
Hey guys,

Made a new video demonstrating the o2 Joggler, and how it works in a LinuxMCE enabled home. Nothing big, just exercising my production chops.

http://www.youtube.com/watch?v=JwZWqv7Jj7c

-Thom

Pages: [1] 2 3 ... 12